StackOverflow programming

Ever since I found this MS/Bing page about internet search engine aided code completion and read about this NSF grant given to Penn for computer aided programming, I wanted to write a post about my friend, John. He is not really called John, but his name is unique enough that he could be identified if I would use it. He is one of the people I would call StackOverflow programmers (they existed before SO but nowadays almost all results are from SO).

John is a professional coder, that is, he makes a living, a very good living, by writing code. Even more interesting, his annual income is a 6 figure number. Very impressive for a self employed coder who doesn’t do any sales himself and depends on his code to work 24/7. His productivity is on a very high level; he manages to write websites,  mobile apps & games at a speed which is frightening, especially for a programmer like myself and especially considering this guy does not know what he is doing when he is writing software.

He does not understand what arrays are, does not, in fact, really understand the concept of variables. Loops are some sort of magic. Classes are convenient, but how they work exactly… Recursion? Well. Who needs that anyway right? He understands some parts of coding, because he can ask questions in enough detail to know what he has to implement and he usually understand about where  to add/change code. I have not found yet where the knowledge begins / ends.

I have seen ‘his code’ and I have seen him ‘code’. The basic process is like this:

  1. Think up what you want to create
  2. Find some tutorial or open source (free or paid) project which does something close to what you need (pro tip: don’t care what language / framework / tech it is in; that’s easier to find)
  3. Find a HOWTO/tutorial on how to install the software you found
  4. Find a video how to install the development environment for the tech behind the software you found
  5. Find the missing features in the existing software; make a list
  6. Take a feature and search on internet, ask the software creator or ask on IRC ‘how to add the feature’
  7. Collect enough concrete snippets to implement the feature
  8. Change the code to match the naming of the code surrounding where you copy/pasted the snippets in (the Bing code completion does that for you)
  9. Create a new StackOverflow feature with the snippets ‘you tried’ and ask on IRC
  10. Go to 6  until done

I have seen him do this with Objective-C, PHP, Perl, HTML, Javascript and CSS.

Unlike most programmers who are well versed in computer science, language theory, functional programming etc, John has been making a small fortune for the past 9 years selling licences and traffic with products he wrote himself while not really understanding most of anything of what he was doing, but rather by using his business brain and a lot of copy/pasting, tinkering and working at a very hasty pace.

Is this a good or a bad thing? As someone who is very interested in formal verification of software and who is afraid of code like this I can only say, yes it mostly is, especially when someone has to maintain the code after the initial coding spree is over. There is a reason for me saying that though.

Before there was Google and most of the internet and definitely far before StackOverflow, I worked with someone who was the predecessor of a StackOverflow programmer, however it was more the ultimate copy/paste programmer. The procedure worked quite differently than searching the internet; in a Turbo Pascal book, he found the basics of what he needed and he manually copied that to the Borland TP editor under DOS. When it worked a bit like what he wanted on one screen, he proceeded to copy/paste that code literally all over the place to make 100s of programs which sold very well and the follow ups still sell well. Containing things like:

  1. a1=1
  2. a2=2
  3. a3=3
  4. a4=4
  5. a5=5

up to over a500, showing, again a lack of what an array is. The programs had to be split up in units and use Overlay because the way of programming took too much memory to compile and run.

When Windows 3.1 and later on 95 appeared on the market, the applications and the then younger me was the one who was going to port them and support them.  Instead of rewriting them I thought, in my infinite wisdom, that it would be better to write a Windows wrapper around the TP sources instead of rewriting everything to something much smaller and more efficient. Big mistake. Supporting so much manual copy/paste labor which was basically puzzled together by only searching in what there already was (a very small StackOverflow the original programmer made himself).

The code bases from John and the other programmer look at lot a like, as do the gaping holes in their programming knowledge. Funnily enough, both do very well commercially; the latter has been doing very well since the 80’s like that. I know at least two other guys who are doing very well in their jobs employing the StackOverflow search procedure outlined above.

Every since I met John, I have been thinking that it should be possible to use Google to do this automatically. But even further than what MS did with Bing. Maybe Watson and Google (to search StackOverflow) as standalone AI + Watson and IRC to get people to make the details work could get somewhere in the future. The question is of course; what would be the input spec to something like that if there is no human to figure that out in the process. Maybe something to that will win me an award in an obfuscated coding contest!

Software development: winning the F1 with a Trabant

Ferrari vs Trabant

Software development is very hard to grasp; it is hard for us, the people who are developing software but even harder for our clients. In the world outside software development things seem to work very differently and that gap seems larger if you have no idea what is going on when developing software. To explain the process to non developers and for non developers talking about software to their peers and back to us, there are unfortunate leaky abstractions of metaphors.

One the worst has been kind of introduced by the industry itself: components. Also building blocks and, when you talk to sales, Lego. Although components in the software sense do not mean what a lot of people think they mean, in day to day language about programming they have been piled on with plugins, portlets, webparts, gems, reusable app and to some extend plain libraries etc to mean that you can take a piece of premade software, put it in your project and strike off another feature in the list.

Sales uses this Lego metaphor during the sales process to get clients to buy products and services which are made by just clicking together a bunch of blocks we already have, preventing both high cost and bugs at the same time. No bugs as the Lego blocks were thoroughly tested at 100s of clients so they cannot contain bugs, right?

As software developers we managed to do this to ourselves with our great optimism. We were and often are the frontrunners in thinking software can actually be made completely composable. We have been making these software tools for non programmers since the 70s and keep making them in the firm believe one day software development will be like Lego. What is worse: we keep telling this to sales and marketing resulting in these great sales sessions.

But then the development starts. And even though these portlets were almost exactly what was needed or the RoR team told actually most of the functionality is actually only assembling a few 1000 gems together, the project is late, buggy, takes a lot more manhours than expected and in the timesheets the word ‘refactor’ seems to appear more than the word ‘the’. How is that possible if we were only glueing together premade Lego blocks?

Even though a lot of architects and developers drink their own koolaid in genuinly believing that this time it will be different, in reality, besides in ‘hello world’ like trivial cases, the metaphor falls flat on his face. Mostly because this abstract software science notion of components has been interpreted wrongly over time and that is getting worse still. So many clients opt for WordPress because ‘everything is already there as a plugin’; why does the project still cost $200k after esrimation and research by more conservative coders and why can a client not just click it together himself?

After the Lego metaphor has been peddled to the client and some sprint build of the project has been delivered, another favorite comes in;

The software has bugs and is not very fast, so when the client, having spent quite some cash on the project, comes in there are some discussions. Usually the agile approach, rightfully so, resulted in a large amount of changes of ‘stuff’ the client did not anticipate upfront.

Lava Lego

 

The Lego didn’t exactly fit and because features are missing, it is anticipated at this point the project will take longer and thus cost more.   This is the time when someone says something along the lines of: ‘when I have a house built, builders deliver in time and for the price we agreed on, how come … … ‘. This is related to the Lego and thus to the components we sold them, so this metaphor is partly our own fault again. But it also isn’t a very good metaphor for the case of the client and yet it is very persistent.

The thing is; houses are not software. Houses are often largely prefab (actual Lego!) and even if they are not, building houses is a rather clear process. And yet, I have known no-one who had a house built were the price was not higher and it did not take more time to built than agreed upfront. Why? Changes by the owner at a late time in the process. Yes, like software development, this doesn’t work well. The metaphor stands quite well if you use it from our side; say you have a house built, but there is no money for the foundation (proper architecture), you are building on swamp land (using flat files in a concurrent environment) and after the entire house was done, you want to add a nuclear bunker underneath (make the application work offline as well as online).

xbridges_down_04-1.jpg.pagespeed.ic.CtcPxU9o0-

Another, worse, metaphor is exactly the same one as above but instead with a bridge. Now bridges are always late and over budget and that is without ‘changes’. And they collapse. So these are projects with people who know a lot about the physics of these kinds of structures and have calculated everything to a tee. And yet it is late and over budget. If they would be the same as software, with the amount of changes (I don’t like the shape of the thing, I want the pillars to be triangular!), it would never finish. Now we are getting close.

Software is very complex and the human interaction which leads up to making the software is very complex; the combination is actual rocket science and makes for this vastly complex landscape which usually cannot be estimated properly upfront as no-one knows what the outcome will be. In projects like houses and bridges, things which are more or less tried and tested repetitive processes, they cannot do it, so how is that supposed to work with software?

This brings us to, for me the worst metaphor; cars. “When I buy a car, I don’t get to pay more so why do when buying software?” “When I buy a Ferrari, I am not satisfied when I drive out with a Turismo!” Obviously this one is the worst as you are comparing a mostly custom product you had developed specially for you with a factory line mass produced one. The issue with this is Lego again; “Chris from sales told us that the process is like sticking the extras on a car with your ‘gems’, so I expect a FXX K with all the trimmings!”

Screen Shot 2015-03-17 at 09.55.49

When the client dictated the price and still has the car metaphor in his mind, you get classics like I recently encountered: a product made for a fixed price dictated by the client, everything else (scope, deadlines) flexible (in the contract), but when the delivery build was done, we got: “But a lot of features are missing here, we need a full featured delivery and we need it now or we’ll sue you for all the damages incurred for not having all those features!”. So I said, more as a joke than anything else; “I buy a 1983 Trabant for E250,- to drive in the F1 and will make the guy I bought the Trabant from responsible for me winning it or not.”, but he understood it now and we have a good business relationship after this.

Since that time I have been thinking more about how to turn bad metaphors into good communication aids with people who are outside of the engineering profession. Working with clients is fun and rewarding, but both sides need to understand what is at stake; software development can be a weird thing for a lot of people. Sometimes productivity falls to almost 0, sometimes it is very high; why? Sometimes a bug takes 1 minute to fix, sometimes it takes a week. The key to this is being transparent and communicating everything to the client and the team every step of the way and documenting every step so it is clear what you are talking about.

 

MonoGame: getting the underlying UIViewController/UIView on iOS (GamePlatform)

For an experiment I needed to touch the internals of MonoGame; more precisely, I required the UIView the current game was drawing on. To do that I checked out the source code of MonoGame and found that if I have a game instance;

  1. MyGame game = new MyGame();

It created an instance of GamePlatform which is an abstraction over the platform specific implementation platform, in this case iOSGamePlatform (to be found in the iOS directory of the MonoGame source tree).

Specifically, iOSGamePlatform has a ViewController method which returns a UIViewController which, as iOS developers know, has a .View property which returns the UIView.

Now the problem is that this GamePlatform (and also iOSGamePlatform) class is not available to you, the creator of games. The GamePlatform is an abstract class which you cannot use from the outside. The only way I could find the instance of this class which is used by the game is by querying game.Services.GetService(), but as one cannot reach GamePlatform, this is not an option.

As I could not find another way of getting to the required instance, I decided to use some reflection to do the trick and it worked quite well;

First, get the Dictionary which holds the instance. This is a private property:

  1. FieldInfo[] fields = bg.Services.GetType().GetFields(
  2.                 BindingFlags.NonPublic |
  3.                 BindingFlags.Instance);
  4. var services = (System.Collections.IDictionary)fields [0].GetValue (game.Services);

Using System.Collections.IDictionary makes for you not having to specify the generics in case you cannot use them.

Then find the proper instance from the services dictionary without actually having access to that physical class:

  1. foreach (var k in services.Keys) {
  2.     if (k.ToString().Contains("iOSGamePlatform")) {
  3.         // found
  4.     }
  5.  
  6. }

And, as a last step, get the UIViewController we wanted to begin with:

  1. UIViewController controller = null;
  2. foreach (var k in services.Keys) {
  3.     // get the iOSGamePlatform
  4.     if (k.ToString().Contains("iOSGamePlatform")) {
  5.         var v = services [k];
  6.         PropertyInfo getController = v.GetType ().GetProperty ("ViewController");
  7.         controller = (UIViewController)getController.GetGetMethod ().Invoke (v, new object[]{ });
  8.         break;
  9.     }
  10.  
  11. }

Now controller.View will have the UIView it is using to render the game which I can now freely use however I want. If there is a nicer method to do this, I could not find it so this hack will do for now.

Edit:

Found a faster way which I initially overlooked:

  1. UIViewController controller = (UIViewController) game.Services.GetService<UIViewController>();

At least I got to teach people about reflection!

Trying out ACE as simple touch display (iPad) editor

ACE

ACE web editor

When I go coding on the go, I currently have a very basic text field on my mobile devices to code in. There is no code highlighting or code completion or anything like that; just a boring text field like Notepad. Now there are (apparently) hordes of people who like to write code in Notepad, but I am not one of them.

The first step to get productive on iOS or Android devices when writing code  is to download a better keyboard. With the standard keyboard, it is really not possible to do much of anything on either device as far as coding is concerned. For iOS I selected ProType ; despite the bad reviews (which are mostly about the condensed style of the keyboard), I found that after a bit of getting used to, it works really well and much faster than the default keyboard for mostly anything, including chat, but definitely for coding.

For Android I use the Hacker’s keyboard which is great; on tablets it’s brilliant, but on phones it does the trick as well.

With that out of the way i’m capable of writing code relatively ok in a ‘notepad’ like TextView, but it’s of course not optimal. So then you have the options to buy some component which does highlighting and possibly code completion for you; there are a few of those but they are not very complete nor very well done. Writing something myself is an option but it will take too much time and it won’t be very robust for months or even years probably. So I opted to try integrating something open source. While it is not a native solution, ACE is a very complete code editor written in Javascript, which seems to run relatively well on mobile browsers. A native solution would have been a lot better, but I cannot find any which are advanced enough to use; Scintilla definitely is, but it does not work on Android and it would take me quite a bit of time to integrate it properly in iOS.

To properly use ACE I would need to integrate it into a WebView to integrate it into my IDE. And that is where it gets hairy. I used ACE before on the iPad on a site which uses a few tricks to make it work properly on a tablet and it was quite ok to work with. At least more than enough to like this as a solution for my IDE. In theory.

Wrapped in a WebView in Xamarin Forms (Github link) for iOS and Android, I believed I had something ok to work with. Then the issues started; obviously the scrolling in the editor is not related to the scrolling in the WebView and as such didn’t actually work at all.

Things like this should make it possible to get that to work on a touch screen, but you won’t get it to work very well unfortunately. I experimented a bit with connecting the native UIScrollView coordinates to the scrollBy() of the ACE editor but that didn’t help either.

The most stable result I got was using;

  1. public AceWebViewRenderer() {
  2. ((UIWebView)this.NativeView).ScrollView.Bounces = false;
  3. ((UIWebView)this.NativeView).ScrollView.ScrollEnabled = false;
  4.  
  5. }

However that was very far from perfect as well. Things that are broken and hard to fix (as far as I know):

  • scrolling
  • double cursor when you have text far to the right (CodeMirror always has this)
  • second delay when moving the cursor by tapping
  • completions like () {} [] are very annoying to work with on a touch screen

Because I lack time in experimenting further and the current results are not very positive, I think I will try to find something else to resolve this. The perfect solution, to me, would be an editor (and keyboard additions) like the one used in Codea; it works well and is robust. I believe it to be fully custom made though which, unfortunately, seems the most satisfying route at this moment…

Still hoping for a better solution…

Writing a game without sleep, power or internet at JFK

Desolate JFK

Desolate JFK

I have a small (…) obsession with programming on the go. With that I mean that I want to be able to write software where ever I am and no matter what I might be doing. Or under what circumstances. So if I don’t have any power or internet for 20 hours I still need to be able to code. This has been one of the things I have been trying to achieve over the past 20 years. My biggest issues have been battery life and portability.

The are kind of related in the end, but to start with the former; batteries came a long way since I touched my first laptop somewhere end 80s. I was slightly spoiled mind you; I had a PC-1211 for years already by then which had seemingly infinite battery life. (Not to mention that I still have it and it still works well) That machine was obviously limited in power, but it allowed me to try out some algorithms on the go. Battery life has been somewhat of an issue though; tablets/phones/laptops all have abysmal battery life. I need days, not hours. And ‘day long’ which has been promised for years now, is definitely not for my days nor the work I do. My new Macbook seldom makes 4 hours while coding.

Laptops aren’t very portable either besides the Macbook Air 11 inch that is (and possible the new ‘Macbook’?), so it is also quite annoying to lug them around especially when I’m off for a 2 day hike in the mountains (and yes, I would like to code while walking). That brings us to portability and that has also been getting better (faster than battery life) but I must say my favorite portable devices are, in order: iPad Air 2 because it’s big enough to do ‘real’ work on, but weighs very little and is fast, Samsung s5 for the same reasons but different work and the OpenPandora 1 ghz version. The last one because it’s just ‘complete’; it has Linux, Android and runs everything I need (sometimes with some effort but we decided to find that fun).

The OpenPandora and the S5 allow me to swap batteries, giving me around 25 hours on the Pandora without any power outlet and around 20 hours on the S5. All that fits easily in a light bag still. For the iPad I bring an external battery which makes everything a lot heavier which is the reason the iPad only comes when I know I need to work on spreadsheets or documents a lot.

Last year around this time I went to the US on vacation; to Orlando to see if Disney World was still fun (it was) and, to get cheaper tickets we had a stopover at JFK from 11 PM to 8 AM. We were a bit naive being from Europe in thinking that JFK is a big airport so there must be stuff open all night and 11 to 8 is not enough time to go into NYC.

After landing we got into a little food court where we thought we would browse the web a bit and have some snacks and coffee until the morning. Unfortunately that was not going to happen. Everything closed at midnight. I still am flabbergasted by that fact and don’t really understand it, but it happened and there was nothing we could do about it. A bit worse; internet wasn’t working on our devices for some reason. So we had no internet, no power and after a while, no lights(!).

I brought my iPhone 5s, OpenPandora and my Macbook Air 11 (I did not have an iPad yet) to watch movies in the hotel or something. As there were no public power outlets, I knew I could not work on my Macbook too much because it would shut down after 2-3 hours (I opened it in the airplane…). Without internet I was a bit lost as I didn’t feel like reading at that time and we started to get tired (different timezone).

By then (around 00:30) I remembered some swiping game a friend told me to try. I did not download it yet, but I played it. Looking it up in the Appstore now I think it was Dots. And because I don’t actually like playing games (I only like making them; I don’t nearly make enough of them though), I decided to recreate that game from just my short interaction with it.

Squares

Squares screenshot

On my Macbook I only had Corona for games and because that is LUA I knew on the OpenPandora I could only use Load81 (as I had nothing else for LUA but a basic installation, and, still, no internet). Load81 is a great tool for teaching people to code as it has no no setup times; you just start it and start programming. Antirez of Redis fame wrote it so it’s like a C=64 from the 80s so people can, again, jump straight in as they could back then, without all the dll hell and compiling, linking etc. Just something to play around. I wrote the basic generators and game algorithms on the OpenPandora; aka, most of the code you see in this file was written in Load81.

After I was happy with how all of that worked in Load81, I used an USB stick to move the code to my Macbook and created the working version in Corona from the demo project. That took only a very short amount of time luckily. My wife and I played it in the Disney World queues most of the time. It’s addictive 🙂

Things that this experience taught me are:

  1. Train your brain; without internet you’d be surprised how much you don’t remember; I was anyway
  2. Downloading and setting up/updating development dependencies is actually far more annoying and time consuming than previously I realized; this was actually one of the smoothest dev experiences in years, despite the constraints
  3. You can finish something nice and addictive in a few hours, so don’t let lack of time stop you; just start coding already
  4. Battery life is indeed great on the OpenPandora
  5. Pulling an all nighter once in a while will focus you
  6. I should (and have been) pursue on-the-go coding like never before (I can code now without internet on the iPad & 5S & S5; more about that soon)

Link to the project in Github. Requires Corona.