Mulesoft “Bug Bounty”

Good afternoon and welcome! Today I wanted to share some of my recent experiences with Mulesoft and how you can use it as a cloud information bus between disparate applications.

For those who don’t know, Mulesoft is a cloud or on-prem platform that not only helps manage APIs, but also helps integrate API systems.

Consider the following: your marketing team has come together and for your annual conference, they want a “bug bounty.” During this bug bounty, Issues can be annotated with the “bounty” tag, and if a collaborator grants the bounty to a user, then that user will get a “point” on a leaderboard.

With that in mind, we need to create:

  1. A GitHub repository with a webhook for Issue Comments
  2. A Mulesoft application that receives the webhook, massages the data, and updates the database
  3. A SQL Server in Azure holding the users and their bounties
  4. A .NET MVC site hosted in Azure to view the database.

Let’s get started!

Before we get any further, we need to check the payload GitHub sends when an Issue is commented on. Fortunately, they have a great documentation on their webhooks API, where you can find their “issue-comment” event documentation.

The full payload is a bit long for this post, so I recommend jumping over to their docs at your leisure. The important point is, we can take the body of the comment, assume it is a GitHub username, and update our leaderboard appropriately.

Remember, this is just for proof of concept. In production, we would need to:

  1. Look for a phrase like “Bounty Granted to: {UserName}”
  2. Ensure the commenter is a collaborator to the repository, or possibly an approved “bounty granter”

Once we understand the payload GitHub will be sending us, we need to turn to the front end and database.

MVC and SQL

Let’s use a SQL server backend with code-first EntityFramework and define our model like so:

Next we’ll create a standard MVC application with a new controller: DashboardController. This will, have one function (Index) which selects the top 10 users in the database ordered by number of completed bounties:

Finally, we’ll make a View that displays a set of BountyfulUsers in a table:

With all that in place, we can deploy our application and database to Azure. Remember, we need to ensure that in our WebApp we have set our connection strings to point to our production Azure SQL server.

connectionString

Now that we have the UI and Database up and running, we need to design our Mule Application to act as a bridge between GitHub and our SQL Server.

The Mule Application

To do this, we need to open up Mulesoft’s IDE, Anypoint Studio, and begin a new project.

The elements we will be using for this are:

  1. Http Endpoint
  2. Transform Message
  3. Database
  4. Set Payload

As you can surmise from the elements used, we are going to receive a payload via the HTTP Endpoint, transform the payload to a more user-friendly data type, update our SQL Database, and set the return value to our user.

At the end, our Message Flow should look like this:

flow

Let’s create a HTTP Listener that listens on all incoming connections, allows POST verbs, and listens on the “/bounty” path. We’ll also use the Metadata tab to set Output Payload to the JSON we found in the GitHub API documentation earlier.

listener

Next, we’ll use the standard Transform Message element to pull out the comment’s body, and pass it to the Database Element.

This is the trickiest part in our application. We’ll need to download the Microsoft SQL Driver jar file from here, and add it as an external library in our Anypoint Studio Mule project. Then we need to create a generic database connection, whose action is to execute the following DDL:

This checks the database for an existing user and increments their BountiesCompleted if they exist, otherwise it creates a new user and sets their BountiesCompleted to 1.

Lastly we’ll set our return payload to success in our Set Payload element.

And with that, we are done with our Mulesoft application! Now we can publish our application to their cloud platform

deploy

Tying It All Together

Grab the URL of the cloud application and add it as a consumer of our GitHub repository webhooks

webhook

With all that done, we can create a comment on an Issue with a user’s name in it, and watch our dashboard update!

theresult

Congratulations!

You can find the Mulesoft XML in it’s entirety (with passwords omitted) here.

Try it for yourself! Grab my code and try adding the extra validation and security I mentioned earlier. Mulesoft makes it easy to configure branching flows and data validation via Anypoint Studio.

Happy coding!

Flutter on Windows without Android Studio

tldr; Install Gradle

Lately I’ve been falling more in love with Dart and Go, two modern open-source languages by Google, and with the recent murmuring around the Magenta kernel and Fuchsia OS,  I’ve been spending more and more time working in both languages (even writing an Epub reader in Dart).

While I’ve been enjoying writing console apps and libraries, I wanted to try my hand at writing some U.I. apps.

Enter Flutter.

Flutter “is a new mobile app SDK to help developers and designers build modern mobile apps for iOS and Android.” and it feels like a lighter-weight competitor to Xamarin. It has a great getting started guide, so I began with the setup.

Note: I’m running Windows 10 Creator’s edition with Hyper V disabled and Xamarin Installed. Your mileage may vary

After git clone -ing the repo and adding the flutter/bin folder to my PATH, I ran the flutter doctor command and got the following:1_flutterDoctorNoSdk

Flutter couldn’t discover my Android SDK (which I had installed via Xamarin previously), which was no problem: I simply set my ANDROID_HOME environment variable and it picked it up.3_sdkInstalled

Android Studio not being installed was problematic for two reasons:

  1. I was on a bad coffee shop WiFi and probably couldn’t download the entire 1.5 GB installer in a reasonable amount of time.
  2. I am a big fan of the Dart community, in particular Danny Tuppeny’s (@dantup)  Dart Code Visual Studio Code extension which makes developing Dart libraries a breeze, so I’d rather use VSCode and his extension over Android Studio.

With those considerations in mind, I decided to skip installing Android Studio and just run

flutter create myapp

4_flutterCreateWorksNoAndroidStudio

Which made a perfectly good Flutter application I can open and work in with Visual Studio Code.5_aWellCreatedFlutterApp

So let’s flutter run !6_cannnotLocateGradeInstallAS

Unable to locate gradle. Please install Android Studio

So that’s what Flutter needed from Android Studio! At this point my download of Android Studio was 50% complete with another hour to go, so I decided to download and install Gradle manually, update my PATH environment variable and give flutter run another try:8_gradleWorksButBadVM

I’m getting an entirely different error now:

Could not reserve enough space for 1572864KB object heap

A quick Google of this instructed me to update my gradle.properties file with

org.gradle.jvmargs=-XX\:MaxHeapSize\=256m -Xmx256m

Now flutter run took me further, informing me that I had not accepted the license agreement for Android SDK Build-Tools 25.0.3. Which was actually somewhat misleading. In fact I had not even installed the 25.0.3 Build-Tools.

A quick trip to the SDK Manager to install and accept licence for the 25.0.3 Build-Tools, and one last flutter run got me to a successfully running Flutter app, all before Android Studio finished downloading.17_runningProperly

18_runningInEmulator

Success!

From here you can iterate and improve on their sample apps or get started with your own!

Happy coding!

Counter-intuitive LINQ

When someone asks me to describe LINQ, depending on their familiarity I might say something along the lines of:

It’s magic!

or

A way of writing SQL-like statements in C#

or most specifically

A set of tools using extension methods and generics to perform queries on sets of data

At the end of the day, however, I do caution them that LINQ is easy to learn, hard to master.

When I first started using LINQ my mentor said “At the end of your query, just send it .ToList(). You’ll thank me later.”

He and I had a few more discussions on why you should be sending your LINQ queries .ToList() and he didn’t know himself other than “Performance and Delayed Execution.”

When working with other C# developers, I find that the Delayed Execution feature of LINQ is the concept they struggle with most. They remember it, work with it, but inevitably write code that forgets that feature, and ultimately create bugs.

Consider the following classes:

Master:

class Master
{
    public Guid MasterID { get; set; }
    public string SomeString { get; set; }
    public Master()
    {
        MasterID = Guid.NewGuid();
        SomeString = "Some Master";
    }
}

And Detail:

class Detail
{
    public Guid MasterFK { get; set; }
    public string SomeDetails { get; set; }
    public Detail(Guid masterFK, string someDetails)
    {
        MasterFK = masterFK;
        SomeDetails = someDetails;
    }
}

Using those two classes, read the following lines of code and think about what the output will be.

static void Main(string[] args)
{
    var mast = new Master();
    var deta = new Detail(mast.MasterID, "");
    var masters = new List<Master>() { mast };
    var details = new List<Detail>() { deta };

    int iterations = 0;
    var joinedValues = masters.Join(details,
                                    x => x.MasterID,
                                    x => x.MasterID,
                                    (x, y) =>
                                    {
                                      iterations++;
                                      return new { Mas = x, Det = y };
                                    });

    Console.WriteLine("The number of times we returned a value is: " + iterations);
    Console.WriteLine("The number of values is: " + joinedValues.Count());
    Console.ReadLine();
}

Got it? Okay, here’s the output:


The number of times we returned a value is: 0
The number of values is: 1

To some of coworkers, when they saw this result, they immediately wanted me to open up GitHub and submit a bug report to the .NET team. They thought they found a critical bug in the LINQ library.

The thing to realize is that in this code we have only created the query when we print out “iterations”, we haven’t executed the query yet, so the value of iterations is still 0. Adding the following line will get results closer to what you expect:

Console.WriteLine("The number of times we returned a value is: " + iterations);
Console.WriteLine("The number of values is: " + joinedValues.Count());
Console.WriteLine("The number of times we returned a value now is: " + iterations);
Console.ReadLine();

Output:

The number of times we returned a value is: 0
The number of values is 1
The number of times we returned a value now is: 1

Since we executed the query when we called joinedValues.Count(), we incremented the iterations variable in our return value, giving the result we initially expected.

A final word of warning on this, however: consider the following code modification. What do you think will be the output?

Console.WriteLine("The number of times we returned a value is: " + iterations);
while (true)
{
    Console.WriteLine("The number of values is " + joinedValues.Count());
    Console.WriteLine("The number of times we returned a value now is: " + iterations);
    Thread.Sleep(1000);
}
Console.ReadLine();

You can probably see where this is going:

The number of times we returned a value is: 0
The number of values is 1
The number of times we returned a value now is: 1
The number of values is 1
The number of times we returned a value now is: 2
The number of values is 1
The number of times we returned a value now is: 3
...

And so on and so on

Every time we are calling .Count() on our IEnumerable (joinedValues) we are re-evaluating the query. Think about what that might mean if you wrote expensive code in your join like so:

var joinedValues = masters.Join(details,
                                x => x.MasterID,
                                x => x.MasterID,
                                (x, y) =>
                                {
                                  iterations++;
                                  //Do some expensive work
                                  Thread.Sleep(10000);
                                  return new { Mas = x, Det = y };
                                });

Then every time you did an operation on that query, you are re-doing that expensive work.

So remember: if you want the code in your join to be executed immediately, or you are doing expensive work you don’t want to repeat, it is safest to send your LINQ queries .ToList() or some other persistent data object.

Hello World!

Hello World!

Need I say more?

I probably should…

Welcome to my first blog post. I’m going to keep it short and sweet because if there is one thing I’ve learned from other internet creators, it is that when first getting started creating content, the price you pay agonizing over creating your first set of content, and painstakingly making sure it is “perfect” in every way vastly outweighs the benefit of the practice and discipline of putting out good content regularly.

So at this point, I’d like to borrow an idea from two developers older and wiser than I: Bob Scheifler and Jim Gettys. Scheifler and Gettys were two of the original developers for the X Windows System, and they had the foresight to, at the beginning of their work, set out seven guiding principles for the development of X. Many of their principles can be applied to all software development, but over the years, one has stuck out to me more than the rest:

It is as important to decide what a system is not as to decide what it is. Do not serve all the world’s needs; rather, make the system extensible so that additional needs can be met in an upwardly compatible fashion.

That is a powerful statement. Limit your scope. Limit your features, but give yourself room to grow. With that in mind, I’d like to define both what this blog is, and is not about

This blog is about:

  • Learning
  • Knowledge sharing
  • Software Development
  • Working as a professional Software Developer and Technologist
  • New Technology and Hardware

This blog is not about:

  • Divisiveness
  • Closed mindedness
  • Dogmatic principles (software or otherwise)
  • “Office Drama”

And with that I leave you hopefully as excited as I am to get started on this journey.