An Overview of the Problem
An increasing number of Plasmoids are relying on web services that we don't control: share (nee, pastebin), weather, public transit, remember the milk, etc. etc. When these services change (which they do randomly, web developers seem to have very little awareness of service stability) our users get stuck with Plasmoids that don't work. We've been working on making the Plasma components that rely on such webservices scriptable so that it would, at least in theory, be easy to post updates that anyone could install on their system. Artur recently made the share DataEngine use Javascript for this, just like the comics and some other DataEngines have done for a while.
Similarly, we've been adding more and more scripting options and features in Plasma Desktop and Netbook. KWin is now quite scriptable (why isn't that blog entry on Techbase as a tutorial? :), and the Plasma Shell Scripting now supports "snippets" that are stored in nice little packages. This isn't unique to Plasma, either: Kate, Amarok and many other KDE applications have also gained similar scripting support.
Problem is: we don't have a good way to distribute these add-ons. There's opendesktop.org, but there are four major downsides:
- Creating a new download type (e.g. a feed for share DataEngine backends) there means making a request of the opendesktop.org crew; they are usually quite responsive, but it's still one more set of human roundtrips
- Anyone can post additions there and there is no way for an upstream such as Plasma to approve select items
- It, understandably, remains a manual process where one has to package the results up, upload them, etc.
- It is driven by non-free software
Even without the last matter, the first three practical matters are not great and probably not easily solvable. opendesktop.org is meant to be a social networking driven community creation site, and it does a great job of that. In this use case, however, that just isn't what we need or want.
My dream solution
So I sat down and sketched out what my dream solution would be, namely:
- A git repository to hold the add-on files
- A simple config file in the repository in which I could define feeds
- Open Collaboration Service compatible feeds automatically created from this, so it can be used via knewstuff (aka "Get Hot New Stuff")
This would make the workflow to start a new feed for the share DataEngine (as an example):
- Edit the providers config file in the repository with:
[org.plasma.share]
compression=zip
packageSuffix=.plasma-addon
metadata=content/metadata.desktop - Make a directory called "org.plasma.share" in the repository
- Commit the addons in org.plasma.share, each in their own sub-directory
- Push the changes to the main repository
Adding a new addon would mean adding a new directory and committing. Updating an addon would mean editing the files and committing. Simple, easy and fits the development workflow we have seamlessly: it's the same way we write any of our code.
Synchrotron
I carved out a few days to work on this idea and finished up the last bits today. I called it synchrotron. It goes with the whole particle physics naming theme in Plasma and sounded like something out of an awesomely bad sci-fi movie. Win-win, really.
You can find the code, currently in what I'd consider to be "alpha" state in my scratch space on git.kde.org. (I'll be requesting a move to playground and starting a review so that it can be moved somewhere more permanent later.)
Synchrotron consists of:
- a PHP script that scans a local clone of the source git repository and builds feeds, packages and a database cache (currently using PostgreSQL) of the providers and content
- a set of pretty simple PHP scripts that power the (minimal) OCS feeds, allowing for listing and downloading only (no voting, no uploading, etc. supported)
All in just a hair over 2,000 lines of code, not that I tried very hard to keep things minimal, though. Huzzah!
Now I just need to get some code review, put it all on a KDE server somewhere, start the source git repository and start having fun.
The path forward
If you found yourself thinking, "I could make that so much better!" or "I could totally write a bit of PHP for that item in the TODO file!", please don't hesitate to get involved. Send me patches, clone the git repo and get hacking .. I am not a web developer and PHP is not my specialty, so I'll be the first to admit that there are problem a bajillion ways to improve on what's there. Some features, like i18n for names, simply aren't implemented yet.
If you found yourself thinking, "I could so use that for my KDE application!" please get a hold of me by email or on irc and let's coordinate. I think KDE Games, KDE Edu, KWin, Kate, Amarok and who knows how many other KDE application teams could benefit from something like this, and I'd love to share an addons repository with all of them! :)
There are some interesting possibilities for improvement in knewstuff, too. For instance, it would be great if the Share Plasmoid could check on startup for addons in Synchrotron and either grab them automatically or offer to. It would also rock to be able to check periodically for updates to installed addons and have them drawn down automatically.
So in some ways Synchrotron is just a beginning, even though its functional already on my laptop and does what I set out for my "dream system". If you're interested in helping out, don't hesitate to clone the repository and get in touch.
Right now, though, I'm off to make some dinner before popping off some requests to KDE's sys admin super hero league to get the Synchrotron spinning. :)
17 comments:
Please please tell me that there is or will be some part of this also called wiggler or undulator?
Keep up the good work, and here's another nut for you - nutcracker :-)
Many open wireless networks only lets you enter a logon page, e.g. this is the case where I work and at the local airport. So you're not really online until you have logged in to the service - whether that requires payment or not.
Wouldn't it be nice if KDE could understand that when connected to network NNN the computer isn't really online until after a login? Then I wouldn't have to accept a load of error messages from KMail, Kopete, the (openSUSE.org) update checker, Skype and what have you.
This is perhaps functionality which belongs deeper in the KNetworkManager stack, but I'll suggest it here anyway since I am not really "in the know" with regards to what goes where.
I understand that from a direct technical perspective the machine is connected once it is connected, but perhaps some brute-force test could solve it (try connecting to some address which will always respond with a known string - and if the response is different, you're not online). Perhaps such a check could even be added as an option to the wireless network settings in KNetworkManager.
Glad to see that bit a bit we're moving away from opendesktop, I like very much the project but it can't be our only (nor main) source of information since it is privative.
Hi Aaron,
thats great to hear.
It´s very good that the Open Collaboration Services are now used like it was planed from the beginning.
It´s not about connecting to one webservice like for example openDesktop.org. It´s an abstraction which allows to connect to different webservices and aggregate the data at the client. The provider files controlling the data sources without the lockin to one single source.
So it´s good to finally see a second content provider. :-)
By the way:
It you use the free OCS server implementation from me (lib_ocs.php at http://www.open-collaboration-services.org/library/) it´s even easier to provide OCS services. You just have to connect the calls with you data source like git in you example.
Please let´me know if you need more features in OCS.
Cheers
Frank
Absolutely fantastic! I've been waiting for update functionality for plasmoids & other scripted goodies for a while now; so pleased to see it materialise :)
The KDE project never disappoints me when it comes to innovation and generally heading in the right direction.
This is excellent! It really does seem to be the ideal solution. Cheers to you, Aaron!
Could it be used for the rest of the stuff in kde-look.org too, like icon sets, KDM-themes and so on?
GHNS is currently impractical because it doesn't work with many of the filesharing sites that people use on kde-look.org.
@te: "Could it be used for the rest of the stuff in kde-look.org too, like icon sets, KDM-themes and so on?"
it can be used for any kind of data, but it doesn't have the user and rating features that kde-look.org provides. this is designed really to be more for upstream (whatever that means for a given deployment :) to publish addons, files and data.
synchrotron would need significant extension to be suitable for user contributions ala kde-look.org.
@kjetil-kilhavn: "Wouldn't it be nice if KDE could understand that when connected to network NNN the computer isn't really online until after a login?"
this would be very nice, if a "detail" item. it belongs in the network management stack, however. Solid could prevent notifying applications of a change in connected state until such authentication happens, but it would need to know of this. it would be easy enough (relatively) to add "needs post-connection auth" to the settings for a given connection, i think the trick would be finding out when the authentication was successfully completed.
i just wish that the companies that create such "log in to a website after you connect" products had worked on a proper answer at the infrastructure level rather than playing games with firewalls and web services. :/
@Frank Karlitschek: "I[f] you use the free OCS server implementation from me (lib_ocs.php at http://www.open-collaboration-services.org/library/) it´s even easier to provide OCS services."
i did look at it, but it looked like it would be more work to figure out how to connect it to the backend and remove all the features synchrotron doesn't need and/or can't provide.
the code for the OCS list/get/download is less than 11% of the total line count.
what would have been very nice is if the xml output was well documented. or maybe it is and i missed the link to it? there are also some odd bits in the OCS spec, such as how Content's download has this:
Arguments: contentid - Id of a content
Arguments: itemid - Id of a content
with no explanation of what the difference is, or if this is just a historical "for compatibility with other or older implementations" issue, and which need to be supported. i ended up supporting both (2 extra lines of code, no big deal).
the 200 reply code also seems over-specified: why spec'd as 15 minutes? can other services have different rate limiting mechanisms?
this is another nice thing about "clean room" implementations: the spec may end up being improved as assumptions that made sense in the origin implementation might not make so much sense elsewhere, or holes in the documentation may arise.
in any case, i haven't yet had time to sweep together my findings from this vis a vis the OCS spec.
Theme song?
Cool. It good to hear that it is easily possible to implement an OCS server. :-)
The documentation could be better of course. I plan to do an OCS sprint in the next 2-4 month to work on version 2.0 of the spec to clean up a few inconsistencies, improve the documentation and implement a specification checker witch makes sure that servers and clients are 100% compatible. This is a request from the Maemo guys.
It would be great if you could join the sprint to that we can put your feedback into the next version of the spec.
Cheers
Frank
Yeah, time to get totally badass names back into IT. Synchrotron. Love it! Don't even care what it does, I just want it :D
Remembers me of my BeOS time where we had Zip-O-Matic and Icon-O-Matic.
Now we just have to reintroduce the "Mega" prefix and the "3000" suffix. X-D
Aaron, we have a automatic script update system using signed packages in amarok. Check it out: http://gitweb.kde.org/amarok.git/blob/HEAD:/src/dialogs/ScriptUpdater.cpp
Great ideas, Aaron, but you could have admitted that some of them had been invented by others ;)
http://lwn.net/Articles/366566/
--
Mark
@markey: hehe. yes. like for example ocs and ghns ;-)
@Stecchino: thanks; i just had a nice conversation with the author of that bit of code today on irc.
i think some of the code there could be lifted wholesale for use by other KDE apps by putting some of it in a shared library somewhere (attica, even?), and the method itself is quite compatible with synchrotron.
@Mark Kretschmann: yes, the idea isn't revolutionary or even particularly non-obvious. i just didn't find an implementation that implemented the concept as i needed it, and i'd. now there is one, and i'd like to share it with others. :)
@Frank Karlitschek: "like for example ocs and ghns ;-)"
synchrotron uses OCS and is therefore compatible with knewstuff (aka ghns). it doesn't reinvent that set of wheels in the least. what synchrotron does provide is an efficient bridge from a git repository to an OCS feed, something that was missing.
Post a Comment