letsthinkabout.us

technology, healthcare, people

Plotting F# Data using Excel Interop: Soaring Life Expectancy Edition

Two centuries after the dawn of the industrial revolution, humanity had its most productive century in the 1900's. The birth of the modern automobile, assembly line, airplane, nuclear bomb, and internet all happened in the same 100-year span. We greatly eliminated infant and child mortality, leading to the first dramatic increase in life expectancy in human history. What are the current trends in this improvement? Let's use the F# Data library and World Bank to find out.


The F# Data library makes it easy to get numbers from the World Bank Data repository. It also features a type provider. If you'd like to read more about type providers in F# Data, see this excellent post.

Let's look for the indicator "Life expectancy at birth, total (years)." We find in that link the key to access that indicator  "SP.DYN.LE00.IN" meaning we can follow the link scheme to access part of the XML data in the World Bank:

http://api.worldbank.org/countries/us/indicators/SP.DYN.LE00.IN?date=2010

<wb:data xmlns:wb="http://www.worldbank.org" page="1" pages="1" per_page="50" total="1">
    <wb:data>
        <wb:indicator id="SP.DYN.LE00.IN">Life expectancy at birth, total (years)</wb:indicator>
        <wb:country id="US">United States</wb:country>
        <wb:date>2010</wb:date>
        <wb:value>78.5414634146342</wb:value>
        <wb:decimal>0</wb:decimal>
    </wb:data>
</wb:data>


In C#, we would typically find the XML schema, write a custom type, parse the data and mine it from there. F# Data takes care of this for you. Even Intellisense exists:



Given that F# is in .NET and can call the same assemblies as in C#, we can use Microsoft Excel Interop to table the data and graph it. F# makes short work of this task, gets World Bank data of US Life Expectancy, tables it, and plots in an Excel graph, all in 35 lines.

open FSharp.Data
open Microsoft.Office.Interop.Excel
 
let data = WorldBankData.GetDataContext()
 
let app = new ApplicationClass(Visible = true)
let workbook = app.Workbooks.Add(XlWBATemplate.xlWBATWorksheet)
let worksheet = (workbook.Worksheets.[1:?> Worksheet)
 
let titles = [| "Year""Values" |]
 
let getDataFrom indicator = 
    let years = Seq.map fst indicator
    let values = Seq.map snd indicator
    (Seq.map2(fun x y -> (x, y)) years values)
    |> Seq.map (fun (x, y) -> [|float(x); y|])
    |> array2D
 
[<EntryPoint>]
let main argv = 
    let murica = data.Countries.``United States``
    let life = murica.Indicators.``Life expectancy at birth, total (years)``
 
    worksheet.Range("C2""D2").Value2 <- titles
    worksheet.Range("C3""D55").Value2 <- (getDataFrom life)
 
    let chartobject = (worksheet.ChartObjects() :?> ChartObjects).Add(400.020.0550.0350.0) 
    chartobject.Chart.ChartWizard
       (Title = "Life Expectancy (All) at birth in the United States", 
        Source = worksheet.Range("D2""D55"),
        HasLegend = false,
        ValueTitle = "Life Expectancy (years)",
        CategoryTitle = "year (0 = 1960)",
        Gallery = XlChartType.xlXYScatterLines)
    0



Our output:




We've increased life expectancy by 10 years since 1960 in the US!


Edit:

The talented folks working on the lower-level happenings of F# have an FSharp.Charting library which I just found out about. Using this library in a .fsx script file, executing with Alt+Enter, I get the same graph even quicker!

#r "../../../bin/FSharp.Data.dll"
#load "../../../packages/FSharp.Charting.0.90.7/FSharp.Charting.fsx"
open FSharp.Data
open FSharp.Charting
 
let data = WorldBankData.GetDataContext()
let murica = data.Countries.``United States``
let life = murica.Indicators.``Life expectancy at birth, total (years)``
Chart.Line(life,Name="Life Expectancy (All) United States")
     .WithYAxis(Min= 68.0, Title="Life Expectancy in Years")
     .WithXAxis(Title="Year")


Result:


5 Pitfalls of C# Beginners

The programmer's brain is vastly different from the average persons'.  However, a programming mind did not develop overnight. As with any other, it is a product of the experiences placed upon it. Learning programming for the first time with C# can be difficult. There are enough shortcuts in C# that simply aren't clear to the beginner. Here are 5 pitfalls.


1. Reference and value types

This is an extremely fundamental part of C# outside of primitive types (int, bool, string, double, etc).  Meaning, if you are writing consumer-grade software, not Project Euler problems or the like, you will need to know this.

Simply put, value types are just values. Reference types point to something in memory. 

The example that comes to mind that was the ultimate point where it "clicked" for me was when it came to List<T>.  I had a List of objects that I wanted to change. I wanted to change that specific list, not a copy of it. Because List<T> is a reference type, you can pass it through a method and you will modify the List you passed in.  Here's the example:

static List<int> numbers = new List<int>(){
    1,2,3,4,5,6,7,8,9,10
};
 
public static void multiplyBy2(List<int> someList)
{
    for(int i = 0; i < someList.Count; i++)
        someList[i] *= 2;
}
 
static void Main(string[] args)
{
    multiplyBy2(numbers);
 
    //numbers now contains values
    //2,4,6,8,10,12,14,16,18,20
}

2. Object initialization syntax

I already prefaced this in number 1. C# unlike other languages has really sweet object initialization syntax. You can set values of properties when you call the object constructor. This should be taken advantage of to its fullest extent.  In my previous example:

static List<int> numbers = new List<int>(){
    1,2,3,4,5,6,7,8,9,10
};
I created the List<int> named numbers and set all values 1 .. 10 inside it. This was done without calling some kind of loop after using new or calling numbers.add(1).. and so on.  Here's another example:

class Drug
{
    public double Dosage;
    public string Name;
}
 
Drug aspirin = new Drug() {
    Dosage = 81,
    Name = "Aspirin"
};

We can still call the constructor if we wanted, but this makes it very clear that the Drug instance aspirin has Dosage = 81 and Name "Aspirin."  

3. Getters and Setters

This is probably one of the biggest sore spots for beginners. It's worse for newbs in C# because of the shortcut syntax. If we extend our Drug example:

class Drug
{
    public double Dosage { getset; }
    public string Name { getset; }
}
This means very little to a beginner. When I was starting out, I was doing Windows 8 metro apps. Since it had a Windows Presentation Foundation (WPF) base, you had to use the get, set syntax to make things WPF friendly. What exactly does the above syntax mean?  It can be expanded out to this:

class Drug
{
    private double _dosage;
    public double Dosage
    {
        get {
            return _dosage;
        }
        set {
            _dosage = value;
        }
    }
 
    private string _name;
    public string Name
    {
        get {
            return _name;
        }
        set {
            _name = value;
        }
    }
}
Stopping point:
  1. Both Dosage and Name are not variables of Drug. They are convenient names for methods that could be named Drug.getDosage(), Drug.setDosage(double _dosage), etc.
  2. We have private variables named _dosage and _name, which is where the respective values are stored.
  3. value is a built-in C# keyword.

Now suppose we do the following:
Drug aspirin = new Drug() {
    Dosage = 81,
    Name = "Aspirin"
};
 
Console.WriteLine(aspirin.Name) //calling get{ } of Name
We indicate that we want the value of Name from Drug instance named aspirin. Thus, the get() method of aspirin is called, which returns our backing value Drug._name.

Now the set() is a little trickier:
Drug aspirin = new Drug() {
    Dosage = 81,
    Name = "Aspirin"
};
 
aspirin.Name = "Motrin";
We are unhappy with the name "Aspirin" of Drug instance aspirin. We want to give it the brand name "Motrin". So we set the Name = "Motrin". The "=" indicates that we are calling the set(..) method of the property Name.

Look more closely:
private string _name;
public string Name
{
    get {
        return _name;
    }
    set {
        _name = value;
    }
}
Inspect, "_name = value" you will find that value  = "Motrin", so we set_name to be "Motrin".

The purpose of these methods is for better encapsulation of your values, a prime object-oriented concept.

4. LINQ

Some people dislike C# for whatever reason. That reason is never Language Integrated Query (LINQ), which is arguably C#'s best feature.

static List<Drug> drugs = new List<Drug>(){
    new Drug("Aspirin"81),
    new Drug("Tylenol"325),
    new Drug("Aleeve"220),
    new Drug("Dextromethorphan"50),
    new Drug("Benadryl"25),
    new Drug("Claritin"10)
};
 
static void Main(string[] args)
{
    var DrugsILike = (from drug in drugs
                      where drug.Dose > 50
                      orderby drug.Name
                      select drug);
 
    var DrugsILike2 = drugs.Where(d => d.Dose > 50)
                           .OrderBy(d => d.Name);
}
As a pure syntax standpoint, there is no need to iterate over the entire collection with a loop, as you would in Java. Here, drugs is filtered based on Dose and sorted by name. Both are equivalent statements, both using LINQ. 

The output for both these using LINQPad is this:



5. The null coalesce ?? operator

We can protect ourselves from using an object reference that would point to a null instance. Traditionally, we can check to see if the instance is null with an if-statement, however we can just use the null coalesce operator (??)


One of the most infuriating exceptions

Example:
static Drug getNewDrug(Drug d)
{
    return d ?? new Drug("Whoops!"0.0);
}
 
static void Main(string[] args)
{
     Drug a = getNewDrug(null);
     Console.WriteLine(a.Name); //"Whoops!"
}

These were 5 pitfalls of C# newbs. Perhaps some at the intermediate level may forget about some of these language features as well. Any suggestions on other things we forget, please leave a comment!


Runge Kutta in JavaScript

There are things you do in life simply for the reason, "because I can." This post is a case of that.

Riding off the coattails of my previous post, Runge Kutta in F# I wanted to know how quickly I could write (copy & paste) it into Firefox's JavaScript debugger.


Total time: 3 minutes.


var x = 0, y = 1, dx = 0.001, target = 1;
 
var sixth = 1 / 6;
 
function rk4(x, y, dx, f) {
    var halfdx = 0.5 * dx;
    var k1 = dx * f(x, y);
    var k2 = dx * f(+ halfdx, y + k1 * halfdx);
    var k3 = dx * f(+ halfdx, y + k2 * halfdx);
    var k4 = dx * f(+ dx, y + k3 * dx);
    return (+ sixth * (k1 + 2 * k2 + 2 * k3 + k4));
}
 
function dy_dt(t, y) {
    return y;
}
 
function Run() {
    while (< target) {
        y = rk4(x, y, dx, dy_dt);
        x += dx;
    }
    console.log(y);
}
 
Run();


I am not particularly competent in JavaScript as of now. I find the loosely typed features to be convenient for this application, and the language lives up to its name: a language where you can quickly script something.

Runge Kutta in F#

I've written how to implement Runge Kutta in several languages now, all that are imperative.

In Java, we had to implement an interface on a class in order to call a function. In C#, we used a delegate, which is similar to a C++ style function pointer. In Fortran, we declared a function pointer using the keyword interface. These are all built-in language functions, but honestly, they all seem unnatural to me.... as if they are some type of workaround, and perhaps not intuitive to a beginner. It makes for a great school assignment to make students pull out their hair.

This post was written while doing an afternoon warm-up for my Thursday 9/25/2014 talk about writing iOS and Android F# apps at Microsoft Chicago.

Again, the purpose of Runge Kutta is to evaluate ordinary differential equations numerically:



A functional language eats this for breakfast, with elegance. This is exactly an application it was designed for. It is almost trivially easy to implement this.

2 things about functional paradigm you need to know:
  1. Tail Recursion
  2. Pattern Matching
In this case, I use both in the same function.  There are no function pointers or interfaces here, since everything is a function!

State your equation:


let dy_dt t y = y
this is the equation y = exp(x)

Use Runge-Kutta 4th Order:

let sixth = 1.0 / 6.0
 
let rk4 x y dx f = 
    let halfdx = 0.5 * dx
    let k1 = dx * f x y
    let k2 = dx * f (x + halfdx) (y + k1 * halfdx)
    let k3 = dx * f (x + halfdx) (y + k2 * halfdx)
    let k4 = dx * f (x + dx) (y + k3 * dx)
    y + sixth * (k1 + 2.0 * k2 + 2.0 * k3 + k4)

Now Solve:

let solve (x:float, y:float, dx:float, target:float) = 
    let rec run x y = 
        match x with 
        |x when x < target -> run (x + dx) (rk4 x y dx dy_dt)
        |_ -> printfn "%A" y     
    run x y
Here's where it gets tricky. We have the recursive function named "run."  Inside of run is the equivalent to the imperative while loop. We match the value of x given the predicate that x < target, we will recurse the run function with new values of x and y. If the predicate of x < target returns false, we match the pattern to the generic _, which prints the value of y

Entry Point:

[<EntryPoint>]
let main argv =
    solve(0.01.00.00011.0)
    0

Result:


y = 2.7184177


which is close to the value e = 2.7182818284590452353602874713527 

As you can see, this is a perfect matching application of a functional language, and it feels just so right and natural.

Here's the full code:


let dy_dt t y = y
 
let sixth = 1.0 / 6.0
 
let rk4 x y dx f = 
    let halfdx = 0.5 * dx
    let k1 = dx * f x y
    let k2 = dx * f (x + halfdx) (y + k1 * halfdx)
    let k3 = dx * f (x + halfdx) (y + k2 * halfdx)
    let k4 = dx * f (x + dx) (y + k3 * dx)
    y + sixth * (k1 + 2.0 * k2 + 2.0 * k3 + k4)
 
let solve (x:float, y:float, dx:float, target:float) = 
    let rec run x y = 
        match x with 
        |x when x < target -> run (x + dx) (rk4 x y dx dy_dt)
        |_ -> printfn "%A" y     
    run x y
 
[<EntryPoint>]
let main argv =
    solve(0.01.00.00011.0)
    0


The Next 30 Years: 2015 - 2045

At any given moment in history, there exists a generational constellation. The mix of generations currently alive and the events that shaped their existence dictates society’s reaction to future events. At times, these events are perpetuated by a single generation, other times will see all generations putting differences aside and coming together.


“History doesn’t repeat itself, but it rhymes.”


The demographics of 2014 America consists of:

Silent Generation (born 1925 - 1942): 

  • Exiting elders born during the Great Depression during low birthrates. 
  • Not old enough to fight WWII, but fought Korea. No one celebrated their homecoming. 
  • They married early. 
  • They were too old to be hippies. 
  • They divorced in adulthood, conformed to new social change, left behind broken homes. 
  • They retired during the 1990’s economic boom, benefitted from it. 
  • Now 80+ years old, society resembles what they grew up with—global-scale conflict brewing amidst economic crisis.

Baby-Boomers (born 1943 - 1960): 

  • Late-adulthood generation who brought about social change in their youth as hippies. 
  • Hailed as the future of America in childhood. 
  • Extremely high birth rates. 
  • Today, their age-demographic is disproportionately large compared to any other living generation. 
  • Rebelled against their elders, initiated a stagnant economy during this rebellion
  • While entering adulthood, gender roles narrowed, drug and alcohol abuse was rampant, and the society their elders built was devoid of spirituality. 
  • In early adulthood, they championed individuality, turned the world inwards and they created the longest recorded expansion of GDP in American history (1990’s). 
  • They stuck it to The Man in the 70’s, they became The Man in the 90’s.

Generation X (born 1961 - 1981): 

Millennials (born 1982 - 2004?): 

  • The generation entering adulthood 
  • This generation grew up chatting with each other over the internet. AOL Instant Messenger 
  • They frequented forums and MySpace, later Facebook, Twitter, Instagram, Snapchat. Now, their community is Reddit, YouTube and just about every aspect of their life is misunderstood by their elders. 
  • Seen as narcissistic and constantly criticized now by a Generation X dominated media.
  • Protected by Baby Boomers.
  • This is a community-centered generation that will soon have a large burden on its shoulders.

Homeland Generation (2004? - 2024?):

  • A generation being born who resembles the eldest generation. Born at a time of low birth rates, economic downtown, and a stirring global conflict pot. 


The current generation constellation is the only valid litmus test to gauge the times. Western society has followed a simple 4-pronged pattern traceable to Martin Luther igniting social upheaval in 1517.


Here, we color the generations by type. As described above, the Homeland generation is a rhyme of the Silent generation. Each era ends in a culmination event, where the generations shift in age. 1967 marked the summer of love and explosion of hippies. 1984 marked the re-election of Ronald Reagan. 2007 marked the economic collapse and 2030 will likely mark the end of our current era. Similar dates not shown in the figure are the market crash of 1929 and D-Day 1945.

 

Describing these eras briefly:


Outer-Driven Era:

  • The world just resolved a crisis. Now, (western) society is of good-feelings and community.
  • Children are overindulged and hailed as the future of the economy and the birthrate booms.
  • Early-adulthood generation conforms to society.
  • The generation in adulthood are not spiritual, and hold the largest span of secular power known in any generational-constellation.
  • Elders are left behind and disregarded, their way of thinking regarded as backwards and not conducive to society.
  • Society homogenizes. But it’s prosperous.
  • The last instance of this was the 1950’s.

Social-Upheaval:

  • Children of the outer-driven era, now in their 20’s find that society shaped by their elders crawls rampant with spiritual poverty.
  • They rebel against the excesses of the previous era, against their elders and usher in social change with a stagnant economy.
  • They break down community and champion individualism.
  • Children born during this era are largely unwanted.
  • Those in adulthood, who conformed to society in the previous era embrace the change being brought in, but because they are too old, they are not the vanguards. They conform again to the new standards and suffer an identity crisis.
  • Those entering elderhood see their secular power crumble before their eyes and watch the constructs of their world burn.
  • The last instance of this was the late 60’s and 70’s.

Inner-Driven Era:

  • The vanguards of change during Social-Upheaval are now adults. They start to resemble their parents who they rebelled against
  • The “losers” born during the upheaval emerge as angry, their culture dissonant.
  • Late-adulthood are silent.
  • Elders are still busy with their civic duty and community.
  • The world turns inwards as people are focused only on themselves—a result of heavy individualism.
  • The birthrates increase and children born during this time are seen positively.
  • There is an overwhelming sentiment: “How long can we keep this up?”
  • The last instance of this was the 80’s and 90’s.

Crisis:

  • An era marked by hardship.
  • Economic and political struggles rampant.
  • Birth-rates are at an all-time low
  • Children born are being over-protected
  • Early-adulthood generation can’t seem to catch a break in society.
  • Mid-adulthood generation have lost an overwhelming majority of their share on society.
  • Elders try to step up as leaders, but are too fragmented on moral and spiritual grounds to be effective.
  • This era culminates in a catastrophic event where the early-adulthood generation must emerge victorious. In instances where they do not (American Civil War), that generation becomes displaced.
  • We are currently in this era.

 

Based on the events of the past 80 years, we are right on track for following this pattern. The Roaring 20’s repeated itself during the 1990’s as a period of unbridled economic expansion perpetuated by an idealist generation (Baby Boomers) centered on individualism. The Great Depression and the rise of totalitarianism are repeating themselves in the form of today’s Great Recession emerging in 2007 and the foreign affairs crisis of 2014. As this pattern persisted then, it will re-occur again, and the outcomes of the world are predictable.

The cycle described above is allowed to continue under 1 condition: that young adults fighting in the crisis emerge victorious. Those who fought WWII came home as heroes and took to public life quickly, without question from any other generation (John F Kennedy). This cycle was broken once in 1865, when the American Civil War ended, there wasn’t truly a victor and those who fought the war became a reactive generation.


Time will tell if I am correct:

 

2015 – 2020:

  • Feminism will reach a legendary event never to be seen again this lifetime. Between 2016 - 2024 is the most likely time a woman will win the presidency in America. If it doesn't happen during this time, no one alive today >20 years old will ever see it achieved.
  • 2016: Wearable technology starts to take off
  • 2017: Internet of Things emerges
  • 2018: US healthcare crumbles under its own weight, brings down massive parts of the economy with it. The nation will be in limbo during this time.
  • Immigration in the US continues to decline
  • Birth rates will stay low
  • Children are over-protected, compared to the 70’s.
  • Global conflict is still brewing in this time period, amidst intermittent illusions of peace and economic recovery. The longer it brews, the more intense it will become.

2020 – 2030:

  • By 2025, a true global crisis will have exploded onto the world scene, not seen since 1939. This can include disease, famine, natural disaster, war and any combination. An extraordinary human cost will be imposed on the world, unfortunately.
  • Late-Millennials (born 90’s - early 2000’s) will fight the conflict. They will win by 2030 and emerge as heroes.
  • Policies set following crisis resolution in 2030 will be the seed for immediate and future conflict.
  • 2028: First Millennial president elected in the US.
  • Claims will be made this decade that humanity has achieved self-aware computing. They will all be false.
  • The 20’s will be seen as an absolute low-point of all humanity up to this point.

2030 - 2040:

  • 2030, self-aware objects will begin to emerge. Technology is community and forges the outer-world as the Millennials see fit.
  • What we see as computers in 2030 would be unrecognizable today. The paradigm will fundamentally change. Computers will reach the capacity of several human brains.
  • Birth rate will explode. Children born in this time will be hailed as the future of the world and will be over-indulged.
  • Young adults, Homelanders, conform to society’s wishes. They will be obedient, subservient, as their life stagnates during this time. They will make life-decisions they will regret 20 years later in the 50’s. If they fight a conflict or assist in a crisis, they won’t have a joyous homecoming, nor will their work be appreciated at this time.
  • Society begins a period of homogenization. People all have similar fashions, all have the same products, all have very similar ideals. Being your own person is acceptable, but you are nothing without the community which will exist in a virtual reality.
  • The economy will be reborn, in a way only Millennials could shape. It will boom, news outlets will describe a new American Dream, different and deemed more prosperous than the old one. Virtual currency use is widespread.
  • Gender roles will widen. Feminism of 1970 – 2020 will be completely lost, and this will be justified due to atrocities seen during the 20’s.
  • Millennials take the House and Senate and their use of technology will be integrated into law. Humans will no longer be allowed to drive vehicles in the name of safety, convenience and productivity.
  • Political discourse now focuses on technology. Do self-aware objects have rights? Is limiting their progress considered inhumane? Should they pay tax? Can they vote? Is erasing their memory violating their rights?
  • Millennials disregard Generation X, who they will see as a relic of the past-- backwards and wrong.
  • Social Security and Medicare are changed in a way where Generation X, now eligible age, will not receive benefits.
  • Generation X’ers lives will stagnate. They will reminisce on times where individuals had power and their life was fast-paced (90’s) and fade out of public life completely.
  • Millennials will begin to purge their own cohorts from communities. Those who were not in the “correct” community in the 10’s will be cast out in witchhunt fashion (ala McCarthyism)

2040 – 2045:

  • The secular world is focused on stability, which it has achieved. The economy is sustained and growing, with some scattered recessions.
  • $1000 of computing equipment will have the processing capability of several thousand humans.
  • There will be a coexistence of human and technology. It will either be integrated into or exist alongside humans.
  • Millennial-dominated government will legislatively side with technology in every possible way.
  • 2045: emergence of self-aware, self-replicating computers.

The events of 2045 will ignite an awakening among 20-year olds at this time. The Millennials of the 30’s leveraged technology to achieve stability, safety, convenience and productivity at the cost of personal liberty. 2045 crosses the threshold. Young-adults will look back at the society of the 30’s as vile, disgusting. They will begin to rebel against the Millennials, much like the hippies did against the GI in the late 1960’s. Their sheer population will be overwhelming and their ideas will overtake every facet of life. Generation X will simply observe as they see the world had left them behind years ago. Millennials will scramble to fight this awakening, but without effect. Homelanders will be too old to rebel, but will conform to new societal change as they denounce their earlier life decisions and pass through a mid-life crisis. The economy will stagnate and there will be a torrential pouring of revolt around the world.

While the most immediate predictions we discussed here are dark and gloomy, please remember this: the last 3 generational cycles (14 generations ago) starting with the founding of America have achieved technology, an economy, a standard of living, and a level of human freedom never before seen at any other time in history. As the world descends into barbarism during this era, it is merely the winter to the glorious spring that will emerge. No pressure Millennials, we just have to win at all costs.