Yeah, I expected it to be tough, this fifth term, what with working two days a week and still doing all the courses prescribed for studying full-time. Still, it looked like it might work. Or in fact like it would work reasonably well, particularly after I had delivered my seminar presentation on functional programming and thus had one course out of five done, except for the being there.
But then the first real practicum assignments for architecture and distributed systems hit us and it was like crashing into a concrete wall. Oh well, to be fair, architecture was and is intense, but nothing out of the ordinary. In a group of six, we are supposed to mock a software company that offers three services interacting with one another, each run by a group of two. The second assignment involved drafting a domain model, process models, and specifications for the use cases (which were defined in the assignment), writing a ReST interface (HTTP communication between services) complete with documentation (Swagger) and tests, and other things like that, so mainly design stuff. We met for a couple of hours in the usual group, and did all that, and then one of us said something like “looks like we’re done”. No, I said, you missed the point “and then implement it”. Which took me another 26 hours of work, 6 of which went into getting the useful, but highly unintuitive Pact framework for consumer-driven tests to work, as requested by the assignment for testing interaction between the three services. No, the task was fairly basic. It just turned out to be a lot of work and hence need a lot of time. And unfortunately, time is my most precious resource this term.
Granted, it was 26 hours mainly because I completed it all on my own, having suggested to my present programming partner that each one of us should do one practicum assignment alone, for a variety of reasons. Primarily because with our fragmented time table, and both of us working, and me having a family, it’s very hard to find time to meet. But also because I found the architecture assignment a lot less scary than the distributed systems one. The latter involved a complicated technical infrastructure and not only does my programming partner find playing with servers, containers, http, ssh and so on far less intimidating than I do, he even seems to enjoy it. And finally, the distributed systems assignment was complex and not easily divisible into work packages. So it made perfect sense to let one person do it. Or to be fair to myself, let one person complete it, because at the point I made that suggestion, I had already invested about a full day of work into this assigment, while my partner, vice-versa, had done virtually nothing for architecture. But since I expected distributed systems to be more work than architecture, that seemed fine.
The problem was that 10 days before both assignments were due to be handed in, architecture was complete, while a week before the due date distributed systems was still more or less where we had left it. And that’s when I started to worry.
And over the next couple of weeks, the distributed systems practicum slowly but inexorably turned into my worst nightmare in this course of studies.
The project we’re working on is the brainchild of the assistant running the practicum. As an idea, you have to admire its creativity. From a distance, that is, when you’re actually not in it. You see, the rule for practica is that the assignments are small, boring examples of what one might do for real with the concepts or technologies discussed in the lectures. For instance, one assignment for operating systems in the third term was writing a token racing car simulation which each car being a thread. That was about six hours of work, 150 lines of code, and it taught us something about threads in Java. Of course it was also deadly dull, but the task was clear and limited. That’s how I like my practicum assignments, after all.
Distributed systems is different. Potentially it’s huge, dangerous, and anything but clear or limited. The assistant is running an environment for a roleplaying/questing game on the server. We are writing an application that enables clients to register with the system and interact with it, and with one another, via HTTP calls, and thereby solve quests. The idea is, as I said, quite creative. The problems are technological obstacles, a lack of regularity or norms in the gaming environment and extremely vague instructions for the assignments, so that solving them becomes inevitably hugely time- and labor-intensive as well as experimental and frustrating.
For the first days we simply had to figure out how to get anything done, at all. To interact with the system, our applications have to run in Docker containers on a certain subnet of the department server. Getting them there involves interacting with a highly confusing GUI written by the assistant that pulls the containers from the Owncloud space associated with our student account, then builds and deploys them, all manually. Every time you change even a single line in your code, you have to click yourself through the whole elaborate process, which adds about 300 per cent to your development time, just to find out if something works or not. Could this have been automated? I suppose. It took us about two weeks to find shortcuts, such as deploying an editor in the container to change the code in place, or accessing the subnet via VPN so we could interact with the server locally from our IDE.
Which still doesn’t change the fact that the ReST interface of the game server is highly particular about the kind of input it expects, which doesn’t combine well with its being sadly ill-documented. For every functionality we implement, we need hours and hours of experimental talking to the server to find out exactly how it works. And every quest, every task, every step seems subtly different from the next, so that writing any sort of interface becomes an exercise in considering every possible combination of events. For instance, solving quests invariably involves submitting “tokens” (lengthy random strings) received after taking certain actions. Yet the submission is anything but standardized. Sometimes you submit a single token prefixed by the ID of the task to a general “deliveries” endpoint on the interface. Sometimes it’s a list of tokens not prefixed by anything, and you submit it right where you are, deep down in the hierarchy of the quest. Which in itself, by the by, doesn’t seem to follow any obvious rule. Some quests have tasks, but sometimes tasks have “next” steps and “todos”, and sometimes they don’t. I could go on and on. Try writing a script, let alone a general application, for that!
For the first assignment we had just a single quest to go by, and since it was fairly simple and straightforward, we basically had a script that solved it automatically. But in time before the actual handing in of our solution, more quests were added, and it was obvious our script no longer worked. So I started working on a general solution that saves the server replies at each step as an instance variable representing the game state, and offers the player options (on the command line) depending on the contents of that state. Say, if the reply contains a token, you get the option to save it locally so you can hand it in later. That was all very fine, and with a lot of experimentation over a couple of days I got so far as being able to solve the second, more complex quest as well.
But then the second practicum assignment increased complexity by about 200 per cent. We are now supposed to group up with other players, exchange messages, accept, solve, and return assignments from our group, even hold elections (mocking the mechanism in which distributed systems determine a new coordinator if the node with that role drops out). All that functionality not only needs to be implemented, but to be integrated in our conditional options menus at the relevant stages. And each option taken leads to others. And there has to be a way back to the previous sets of options, necessitating some creative logic considering all possible ways through the menus, and a stack of game states to return to. A positive nightmare. Maybe we should have stuck with a simple command line script. But after all the time invested in this much nicer solution, there seems no way back. And of course the two options don’t combine.
I wonder what kind of escalation the third and final assignment can bring?
Apart from the sheer size and complexity of the project, the main problem is unclear instructions. The second assignment is a four page wall of text, and after two hours of intense discussion with my programming partner I confidently thought we had considered and defined every aspect of the many procedures involved. Yet once I started working on my side of the problem (this time it was reasonably divisible, with me writing the client logic and my partner the persistent service that handles communication with other players), it became clearer with every step that even those four pages leave a lot of detail open. In fact, by my estimate fully half of the necessary details are missing.
We talked with the professor and bombarded the assistant with questions per email. Both claim that uncertainty is of great didactic value, for in real life a software designer also has to deal with great uncertainty, with requirements being unclear, and systems design offering a great amount of leeway for different solutions.
Yet while uncertainty is alright if you design your own standalone system and can creatively exploit it, it becomes a positive liability when you are forced to interact with others. In short, if we are to communicate with the game server and other game clients, the ways of doing so–say the formats of the JSON strings we exchange–must be properly defined by a central authority. Interaction doesn’t work if every group designs their own standard. A distributed systems without standards is a joke.
Worse however is that this amount of uncertainty, combined with the technical difficulties, increases the necessary time effort, for one practicum assignment out of three for a single course out of five, out of all proportion. We simply don’t have the time for this kind of experimental design process. I know I don’t. Working two days a week in the real world, it’s hard to find even a half-day per week for anything studies-related, and the practicum assignments and the studying time for all four remaining courses compete for that scarce amount of time. I admit I somewhat sympathize with the “uncertainty is good” theorem, but that still doesn’t justify practicum assignments whose complexity, vagueness, and size are all virtually unlimited. For instance, the quests could be more streamlined and the functionality for group interaction more limited, and we would still learn the same lessons and demonstrate the same skills, but in a fraction of the time needed for this over the top nightmare. Uncertainty is OK, but there must be limits.
And it doesn’t really help that the professor and the assistant both hinted that we would get the PVL no matter the results, if we just demonstrated that we had banged our heads against the wall hard enough. The workload and stress in this course of studies is very intense at the best of times, and not knowing whether one will be able to complete a practicum assignment is the purest form of stress I can imagine. Fail in the fifth term? Fail to complete the requirements for the good job I am virtually certain of? For me personally the uncertainty over that in the past couple of weeks has been nearly enough to drive me back into depression. At times I have trouble breathing.
And one of the worst things is being dependent on my programming partner. For about a week before the first assignment was due, I wasn’t sure whether he would ever do his part of the work, and if, then when and how well. About the worst scenario I could imagine was our sitting there and not having anything to show. In the two nights before the practicum meeting I hardly slept because I didn’t know what my partner had done or was doing, except that from what I had seen it wouldn’t suffice to meet the requirements because so far the script didn’t even solve a single quest. In the end, we just about passed, but what if we hadn’t? There is no chance I am doing these insane assignments on my own, so I need him.
But it doesn’t bode well. After the discussion we had on Monday when we defined the interfaces, we agreed to meet again on Wednesday to try and see if our two services worked together. I skipped work on Tuesday (I still have an insane amount of overtime) and did my part of the work, more or less all of it, but had no way of testing it without access to his service. On Wednesday we sat side by side, but he hadn’t implemented even a single functionality yet. For two hours I watched him fight with HTTP calls and tests and thought “we’re wasting our time here”. You see, if it’s so hard to find time to work together, it should be obvious that when we are actually together in the same room we should be doing things we can’t do alone. I finally urged him to at least see if we could couple our services in some way, functionality be damned, so we knew at least we were on the right way. He grumbled, but then came around, and now, less than a fortnight before the due date, we at least know that we can exchange messages. Which should be about 10 per cent of the functionality required for the second assignment. Oh never mind, I have implemented all of it on my (the game client) side. But it won’t do us the least bit of good unless his service provides the communication with other players. And once he does, and only then, can I start debugging my side which was really written in a fit of blind flying. So I am again, totally and inexorably, dependent on him doing his share. Which didn’t work so well the first time. I suppose I’ll have trouble breathing for the foreseeable future.
Of course the third assignment for architecture is out as well, which involves deploying our respective services to Docker containers and then writing integration tests for them. My programming partner volunteered to do that. Which is all fine with me, if he really finds the time to. You see, we are doing container stuff in three courses out of five (apart from these two, our neural networks project uses containerized services as well), but I haven’t been able to find any time to acquire the necessary routine so far, after having done the Docker tutorial (yet again). For when?
I am also behind with my flash cards in a way I never was, in four terms of using them. Right now I have 175 new cards for architecture and distributed systems combined–cards I haven’t seen since writing them. That’s in addition to the 200 or so old ones Anki thinks I should urgently review. Basically I should memorize 20 new ones, per subject, every day. When? And every architecture lecture adds about 60 to 80 new ones. We are now in the second half of the lecture, read by a new professor, and he is very fond of presenting insane amounts of material that probably even he would admit, if pressed, is not entirely relevant for the exam. Like every notation element of 16 kinds of UML diagrams. Or the details, pros, and cons of two dozen architectural styles, at least half of which have been obsolete since about 1970. I have no idea when I will find the time to stuff all that obscure knowledge into my brain.
And what about this blog? Today I thought it was about time to get off the hamster wheel for a couple of hours, for the speed with which it’s spinning right now is driving me nuts, but that’s the first time I’ve written in what, three weeks?
I probably shouldn’t even mention that right now we’re also looking for a new house. Ours is simply too small for five people, three of which will be teenagers within a few years. When are we doing that? When will we move, should we find one? Don’t ask. In fact, as the kids grow older, they seem to need more of our time rather than less. True, we no longer change diapers. But the big one is taking drum lessons. The middle one is attending swim club two days a week and actually should sing in a choir, for like the girl in “Thank You For The Music” she simply can’t help singing (what do you want? I grew up in the time!). And the small one has started gymnastics training. And even though we live in a city, all these kid activities involve taking them somewhere and then waiting around with nothing to do. Oh well, there is flashcards on the cell phone I suppose.