The Last Race for SuperApps

SuperApps have had great success in the Asian markets, but not so much in the West. Make no mistake, the tech leaders in the US markets have the same objectives as SuperApps – connecting your friends and lenders alike, they want to be in the middle of the every transaction – emotional or financial. Yet, there are a multitude of reasons behind no SuperApp has emerged in the Western markets. As Generative AI matures, all that could change very fast.
If you haven’t noticed, the apps that dominate our daily lives have been smoothly sliding LLMs into our dm-s. Google rushed Bard out of the door when they saw ChatGPT, Microsoft has Copilots ready for every enterprise offerings, WhatsApp got Meta AI, SnapChat launched My AI, X recently got Gork, while Alexa and Siri are now the boomers in this market. Below the veil of chatbots, this has reignited the race for SuperApps. What is different this time? How would the SuperApp look? And why would this race be the last?

Chat as the ultimate UI

At the core of any human computer interaction we have, Input -> Compute -> Output. It has been a long road to reduce the friction at the boundaries of compute – making computers comprehend our intent and respond back meaningfully. For decades we have been stuck with the graphical user interface – which at best provides clunky forms with restrictive UI patterns.

But standing in 2023, that friction has been smoothened out considerably by LLMs – our natural language is very close to being sufficient to instruct the computers. On the output side, we no longer need to wrestle to make the computation results comprehensible – we can get natural language description of the compute at the right level of compression that reduces the cognitive load for interpretation.

What could the frontend of our apps look like now? Does an average white collar worker need to click though multiple tabs of internal and external websites to search and sift for a needle in a haystack? Or could they just ask for the information to be collected and summarized on their chat app, while focusing on making better decisions? Does the busy parent need to click through a between multiple tabs of calendars, email, shopping websites to figure out how to plan the week’s dinner, month’s budget, carpool schedules? Or chat on their favorite social app with a bot that understands the complex dynamics of the modern family letting the parent build fulfilling relationships?

Have you noticed how every non-conversational apps like your bank, your brokerage firm, or utility company now has a barely functional dumb chat client plugin on their app or website? There could be hundred more examples, but the pivotal example is that of ChatGPT. The most successful product launches in history – just a simple chat UI with a single textbox, and a few buttons. All the GUI elements, which are really proxies for natural language have melted away in the face of direct conversation which needs no user training.

This is why conversation first apps like Slack, Teams, Twitter, Whatsapp, Meta’s messenger (Meta AI) are well placed to become SuperApps both in the office and home front. Most consumer facing apps will evolve to become rag (Retrieval Augmented Generation) apps as they ditch the whole click driven UI to become chat first UI. Or, they will melt into one of the potential SuperApps. Just like every company was getting on the web in the early 2000s, and then on the mobile App Stores in the last decade – sometime down the line dedicated websites, apps for individual establishments could become a relic of the past. It would be much economic for many organizations to just have a Custom GPT, which can access the shared data and uses the chat interface of the SuperApp. The question is not “if” or “when” – but really, “where” will that user interaction happen? How will the SuperApp contenders need evolve their platforms to win the trust of the consumers and industries just like App Stores did a decade back?

Move Right

Moving from the user interface to a lower level of computing, we have the compilers or interpreters that take pseudo english programming languages and convert them to zeros and ones for the processor. With natural language being the ui, do we really need intermediaries like a programming language, a compiler?

GPT plugins, specially Code Interpretor, have already shown us what just in time computing can look like. Stretch your imagination a little further where integration, implementation, testing and execution all gets done in real time as response to your prompt. The fact that LLMs today can generate quite accurate code at a speed acceptable to many real-time scenarios is the second big opportunity for creating SuperApps. Niche business logic that earlier took months to code, test and ship can now get autogenerated, compiled and executed – all on the fly based on the user prompt.

Dynamic code generation and just in time execution with the continuous re-authoring of complex business logic based on user feedback will be key factors in the race for SuperApps. As the landscape will go through tumultuous disruption, the pace at which we write code and ship software will no longer be sustainable. Current development best practices say all development, testing etc needs to be moved left. But my guess is that as LLMs get better, we will see an absolute paradigm shift where the entire software development life cycle moves all the way to the right, towards the edge. The models in the SuperApp will eat and regurgitate the software stack into a completely new beast that focuses on quality, safety and security.

Data – the only moat

With the user interface and computation leaving less room for competition, it all comes back to the age old wisdom: the one who keeps control of the data wins. The absolute key battlegrounds that every app is fighting for and building moats around is data. Both historic data and real time data. In the last decade we saw how tech companies begged, borrowed and stole user’s data to build billions with personalized recommendations. This gives them massive advantage in terms of training their closed source models with all forms of human interactions. That base paves the way for Copilots or CustomGPTs to be the next generation of personalized recommendation.

As these tools supercharges content creation on these platforms, so will the monetization of it. GenZ knows this. They are way ahead of the curve in generating content on these SuperApps contenders, that celebrate and reward their unique perspective to captivate attention. They know media entrepreneurship will be the most common profession of their generation. They know their data will work for them while they sleep. The question here again is, which platform will they choose with their data and content?

The last race

On the spectrum of market capitalization, network effects, advanced models, partnerships, and product offerings, contenders like OpenAI, Twitter/X, Google, Meta, and Microsoft vie for supremacy. Each brings unique strengths to the table, making the race for SuperApps an exhilarating competition to watch unfold. However, the competitive advantage that each of the players has, could be disrupted in a matter of days obliterating empires. Whoever has the correct combinations of technology and market fit, will attain escape velocity before others can catch up. Case in point, the never before seen user acquisition by ChatGPT – 1 million users in 5 days. Recall this February, Google’s drop in share prices by 9% after Bard’s demo had mistakes in the response, home work helper Chegg losing 50% of value due to ChatGPT’s four months of existence. We have no clue on how to regulate the power that hides behind the innocuous interface as it flywheels into a monopoly.

It has become clear that the success of the AI arms race is biased towards deep pockets. Our only hope is that in the past, we have been successful in building open technologies that benefit the whole civilization rather than for profit companies. Take the Internet for example which triumphed over the proprietary Information Superhighway that Microsoft wanted to establish in the 90s. Open Source softwares like LAMP stack that got the world to its digital adolescence. Or Bitcoin or Ethereum are hallmarks of decentralized technologies. We need open standards, protocols and software for building and regulating models. That way when singularity unfolds, it does not happen behind the closed doors of multibillion dollar corporation, but out in the open with the whole humanity at its core.

With proper regulation, open protocols, open weights, may be we have a chance that we don’t end up with the dystopian future that with AGI. We will not be in awe of harnessing it. We will not be bothered about “where” it comes from, “how” to make it work with the world around us but take it for granted that it works for us. Instead, computing will fade into the background allowing us to maximize our unique gifts to build something new.

A (Post-Pandemic) World Without Email

Stress induced by unread messages is an undeniable part of modern inbox zero work culture. In a post covid hybrid/remote first world, the trinity of email, Slack and Zoom has become the default knee-jerk reaction for workplace interactions. The lack of physical presence has exacerbated the expected response time of asynchronous messages. While messaging apps are adding LLMs for auto generating responses in seconds, I am here to sing the praises of a book that wants to remove email altogether.

A World Without Email: Reimagining Work in an Age of Communication Overload, was released right in the middle of the pandemic when remote work was at its peak worldwide. The key message of the book could not have been more timely:

Unscheduled asynchronous messaging as the default backbone collaborative knowledge work is detrimental to the quality of output.

What follows are some of my realizations in the process of incorporating the principles from the book. I became convinced that for building high performing teams for collaborative knowledge work, we need a culture shift. And that can happen only when organizational leadership commits to workflow design with empathy, instead of incessantly chasing work execution.

Knowledge work – The Workflow and Work Execution

The book’s central thesis revolves around the idea that there are two facets of knowledge work: workflow and work execution. The growth of white-collar jobs during the economic boom of the last century has led to a substantial portion of the workforce engaging in knowledge work. However, due to its intangible nature and rapid evolution, there has been a significant lag in the development of efficient workflow design. The lag was further impacted by the digital revolution, specifically the emergence of email and chat as ubiquitous communication tools. These technologies facilitated lightning-fast message delivery across the globe, resulting in widespread addiction under the guise of collaboration.

This short-sighted view on immediate results neglected the long-term consequences of email driven work culture. Computers and networks kept getting faster, but the human at the either end of a message did not. They jumped from one unfinished task to another, with an illusion of multitasking like the computer. Busyness became synonymous with productivity.

Cal’s manuscript was probably done before the pandemic, so there is no reference to work during lockdown, but we all know what happened. Our lack of preparation for pandemic led to a global spread of the deadly virus. Much like that, when the lack of efficient workflows met with an abrupt switch to remote work, it left the knowledge workers inundated by non-stop Zoom, Slack and emails.

Refocus on Workflow

Any task in a collaborative knowledge work environment goes through a set of state changes – inception, discovery, refinement, work in progress, execution, waiting for feedback, completion. Without a predetermined protocol, every task needs to establish the next steps to completion through unscheduled asynchronous messaging in the form of an email or an instant message. The book describes this chaotic state of collaborative work as the hyperactive hive mind which eats away the attention capital of the organization. The open loops of unfinished work linger long after designated work hours as people struggle to fall asleep while scrolling through unread messages on their phones.

Jira, Asana, Pivotal Tracker, Trello – there is no shortage of tools in this realm of ticket based workflow design. If you are already using one and not getting the benefit, ask the following questions:

Is everyone in the collaboration ecosystem is onboard to using it?
Are the list of tasks gets reviewed at regular intervals to evaluate progress against deadlines?
When new work or uncertainty surfaces, is there a knee-jerk reaction of shooting an email/instant message or opening a ticket?

Transitioning from email to workflow tools in five steps

Take a step back and try to understand the complex patterns in your team’s work.

Start with a prioritized queue of tasks needed to take an idea to completion
Define the different states a task can be in. Start simple ToDo, Doing, Blocked, Done. Don’t over-engineer, stay between three and five.
Have a predefined protocol that determines who does what at each state transition
Have a shared view of who is working on what and which state the tasks are on
All asynchronous communication for discovery, decisions clarification, execution, feedback etc. happen in the context of the task

Role of leadership

The role of leadership buyin is the linchpin here. Once they see that chasing the stream of consciousness from the hyperactive hive mind is not the best collaboration strategy, it becomes straightforward to establish a culture that translates work into meaningful actions within the framework of tools and processes.

Work Execution Modes – Specialist and Support

To gain something valuable like autonomy, you have to offer unambiguous value. You have to be accountable for what you produce, if you want the freedom to improve how you do so.
– Cal Newport

In typical knowledge work, contributors have the opportunity to deliver value in two distinct modes. The first is the specialist mode, in which they can immerse themselves in deep, solitary work. The second is the support mode, which involves engaging in collaborative problem-solving. When the support mode is neither scheduled nor governed by structured protocols, it can easily transform into a source of incessant distractions, thereby impeding the performance in the specialist mode. Meetings, whether in person or remote, are the bedrock of support mode and possibly is the second most hated artifact of collaborative knowledge work after emails.

In today’s hybrid work environment, it’s crucial that we foster a cultural shift towards empathy, avoiding unnecessary calendar clutter for our colleagues. It is inspiring to see some organizational leaders championing ‘meeting-free quiet days’. Yet there remains ample room for improvement. Hopefully in the future, calendars will not only check for common availability, but also factor in the priorities of their deliverables and the cognitive burden of the employees to avoid burn out. Till then, here are some pragmatic approaches to calendar management.

Collaboration hours

Proactively providing designated time during the week as collaboration hours cuts down adhoc exchange of messages for agreeing to a meeting time. Anybody seeking your time for any collaborative work like knowledge sharing, discovery, troubleshooting etc, can use these slots. Outlook, Calendly, Gmail all provide options for booking time and even encourage putting them in your mail signature/chat client status.

Standups

In software teams, a common practice is to conduct a daily standup meeting. During this meeting, team members take turns summarizing their accomplishments from the previous day, setting objectives for the current day, and addressing any obstacles they may face. When conducted with the right mindset, this can be the most effective meeting a team can have.

These meetings are called ‘standups’ deliberately to keep them short. If a standup involves more than six-seven people or takes longer than ten-fifteen minutes, it has likely become unproductive. Any other topic otherwise is discussed in a ‘parking lot,’ where only needed team members break out to address specific obstacles, streamlining the main meeting.

Ad Hoc meetings

While the above methods significantly reduce the necessity for impromptu meetings, they cannot entirely eliminate them. Structure the agenda as a list of questions on a shared document, each with possible solutions, and convert meetings to decision-making process. The outcome of the meeting are action items for designated team members with deadlines which should get reflected/tracked on the workflow tool described above.

Other experiments

While that utopian day of the world without email is still far out. Personally, beyond the rules for auto-deleting and auto-archiving, those email that contain information but need no actions I manually push then to my Projects/Areas/Resources/Archives folders in my second brain using OneNote and Outlook integration. I never touch them again in Outlook, but progressively summarize them in batches.

Emails that need actions I forward them to Jira, using email to jira integration. Those that absolutely need replies, I try to shut down those threads completely with super short replies containing hyperlinks to jira tickets, wiki pages, shared documents with checklists and instructions. It is a game – if email replies exceed five sentences, I reevaluate what more can be done to liberate the task out of email entrapment. To reduce slack messages, I’ve been experimenting with Canvas where we keep a list of open items to be reviewed during scheduled meetings rather than defaulting to adhoc pings.

Conclusion

Email’s impact on the workplace was not additive; it radically transformed the very fabric of knowledge work and our professional lives. In the wake of the pandemic hangover, our society finds itself grappling with the cognitive load of increased digital communication, all while navigating the pressure of return to office. Let’s be more intentional than adhoc to build a future where the distraction of unread messages, or the anxiety of their accumulation, no longer dictates our capacity to deliver our highest quality work.

Audacious Aspirations of an AI Optimist

According to Challenger, Gray & Christmas, Inc.’s report last month (May 2023), 3,900 jobs were lost to AI. That makes it the seventh highest contributing factor for layoffs and 5% of total job cuts of the month. This is also the first time AI has been cited as a contributing factor for job loss.
AI eating all jobs doomsday predictions dominate the mainstream media. Our news feed are overflowing with predictions of societal collapse and existential crisis. There is no way to leapfrog the misery of job loses in the coming years. Nineteenth century definition of worker, job and career have reached their expiry date.
But there is another side to the story. New definitions of jobs and careers for the next hundred years are getting written today. And they spell an era of unprecedented economic boom beyond the temporary misery.

The last century

With every breakthrough in technology, we have seen the old jobs and careers vanish. New jobs and careers open the door of opportunities by orders of magnitude. When goods hand-crafted by artisans got replaced by those from the mills and factories, it brought thousands from rural areas to fast growing cities. When internal combustion engines replaced horses, the coachman gave way to thousands of truck drivers who criss-cross the country everyday.

With software eating the world, we have seen people in a variety of new jobs and careers which were unimaginable to their parents. Those with the economic affluence to afford a good education saw a path to flourish as knowledge workers. Those less fortunate became mechanical turks in the gig economy serving just in time needs of the affluent at a narrow margin.

Unfortunately, during these years of progress our view of economic growth became centered around the success of big corporations. This twentieth century model has succeeded by leveraging centralized capital at the cost of human resources. Those in possession of capital enriched themselves, shaped the values of our society, and made participation and success a struggle for the less fortunate. Thus despite these advances the economic rift that bisects our society kept widening.

In the last decade, three new forces have emerged. They are on the verge of alignment to bring the biggest change our civilization has ever experienced.

1. Influencers

Influencers are independent individuals who through their originality and creativity are able to impact the outlook of thousands, sometimes millions. True fans in niche communities are evaluating their content and coronating the deserving creators. They understand how to leverage the new technology and media and ditched the 9-to-5 grind from their parent’s era.
While this rise of individuals was promising, it did not loosen the grip of the big companies. These influencers are chained to the algorithms of the platforms where they publish their product or content. The algorithms control every aspect of a creator’s success and are written behind closed doors to maximize the profit for the companies. The creator and their fans got trapped in the walled garden.

2. Decentralization

The advances in decentralized networks and payment technologies has made it possible to break free of the network effect these platforms. These solutions can go beyond the control of a single company or censorship of a government.

Recent debacles in the cryptocurrency world have surely shattered the confidence and led many to abandon ships. But beyond the hype cycle, the technological breakthroughs that we have already achieved are here to stay. Once they mature to provide the safety of the financial assets and protection against censorship, it will give the creators freedom to focus on their craft and community.

3. Copilots

The third and the final force are AI copilots. The sharper your knife, finer is the cut. With generative AI, we are in the early days of creating a general purpose compilers for building new tools.

The current generation of AI tools have given millions the power to rapidly remix the knowledge curated by humanity over ages. These tools excel in visual, linguistic or computational expressions and can create something that shocks even the most proficient practitioner of the craft. Never before has such a technology been made accessible to human kind at such a scale.

As AI super charges our tools, we will create higher quality results in record time. Our personal AI copilots will evolve with us from every meeting we attend, every email we are cc-ed, every new book or article we read. They will act as a mentor, a career coach, keep our inbox zeroed, and maintain a prioritized list of where to focus on for maximizing our goal.

Humanoid robots managing manufacturing, self-driving distribution fleet delivering goods, a giga factory at a natural language api call away – the grunt work will take a back seat. The focus will be on creative work.

We can imagine four stages of this creative work

Stage 1: identifying a problem and imaging the solution for a community

Stage 2: prompt engineering the implementation

Stage 3: review, feedback and course correct till a steady state is shipped

Stage 4: maintain steady state cruise control

Employee to entrepreneur

In the future, a typical workday of an employee will probably look similar to that of today’s entrepreneur. They’ll act as the conductor orchestrating the autonomous agents by providing higher level decisions and strategies.

Employees thinking and acting like entrepreneurs will change everything. More and more people will find their niche community of true fans and realize their true potential outside the confines of a company. The promise of a life long job and career with the same company will no longer be the norm. Free-lancing, contracting, or consulting will become more dominant model of how people earn their livelihood. Serial entrepreneurship will become the de facto career description in the coming days. Salary to equity transformation will help bring more people out of chasm of poverty.
Quality, originality and creativity will be rewarded than winning the race to be the cheapest. People will be paid for managing the tension to establish steady state control to any problem facing disruptive forces. But the rate of disruption will only accelerate, setting a flywheel effect of unprecedented economic boom.

Weakening of Big Corporations

The success of open source AI development shows how this dream can become a reality. Closed source models have exhausted billions of dollars to define the state of the art. However, in the open source world, new models get published everyday. Their active leaderboard of performance is an existential threat for the big companies – they have no moat. Open source attracts motivated individuals who can learn, teach, build, change and share without permission. That pace of innovation is impossible in the high stake rigid environment of a big company. This development is backed by organizing global community of true fans. They are contributing with human, computing or financial resources in foundational research, implementation and their usage in real world problem solving. Such development will make it harder for traditional big companies to compete and sustain profitably.

Open Regulations for AI

Great power in the hands of the irresponsible is a recipie for disaster.

Regulating AI and figuring out the real world consequences is the challenge of our generation. This is why leaders from big companies and countries are holding emergency meetings. They are scrambling to come up with a playbook that will allow them dictate the ecosystem.
But we cannot rely on them to call the shots with this powerful technology. We have seen too many examples of how the society and nature have suffered when power is centralized behind closed doors. We have to demand laws and regulations to be written, reviewed and modified in the open and driven by the community. Progress in blockchain technologies have laid the foundations for decentralized governance, policy making and enforcement. Cryptocurrencies and smart contracts have shown successful implementation is possible. Whether we will be successful in creating an economically just world will be dependent on our success in effective democratization of AI regulation.

AI Optimist

If you have made this far, I am grateful for your attention to this dream.

I hope the rise of the individuals and weakening of profit driven mega corporations will see the demise of other old societal structures that keep us from achieving our true potential. Decades down the line, we will find the rift between the rich and the poor has shrunk as we continue to move towards a more just society driven by service rather than greed. An era where policies are written to help human species to flourish in this planet and beyond, staying in harmony with nature, will finally begin.

Data pipeline for vacation photos

I take pictures when I am on vacation. Then I throw away 90% of them, some make the cut to end up on Instagram. Instagram is a great platform, however without an official API to upload images, they make it tough for lazy amateurs to publish their content. There is a popular unofficial api, which I intend to give a try.

But even before I get there, I need to get the pipeline ready for getting the data out of the memory card of my DSLR and finding the ones that are worthy of posting. I don’t do any editing what so ever – and proudly tag everything with #nofilter. The real reason is image editing is tough and time-consuming. I doubt anyone would have such a workflow, but I find the existing tooling frustrating makes me do boring manual jobs – that too on vacation.

The workflow

Typically when I am on vacation, I would take pictures all day and as soon as I reach the hotel I want to get the pictures off my camera, group them in keep, discard,maybe buckets, upload them to the cloud, post them to Instagram and finally have the memory card cleaned up for the next day. If I get about an hour to do all this – I’m lucky. The most time-consuming part of the workflow is looking at the images and deciding which bucket it belongs. Rest of the parts are easy to automate. So I wanted to take up that part of the workflow first.

This stage of bucketing a few hundred images into keep,maybe,discardbuckets needed a tool that is more flexible than Photos on mac. Sometimes there are multiple shots of the same subject which needs to be compared next to each other.

After some digging, I found feh. It is a lightweight image browser and offers productivity and simplicity. Installing feh was fairly simple – just install it us if you are on a mac.

brew install feh

Feh acts like any other simple fullscreen image browser,

feh -t # thumbnails 
feh -m # montage

Other useful keyboard shortcuts

/       # auto-zoom
Shift+> # rotate clockwise
Shift+< # rotate anti-clockwise

There are tonnes of other options and good enough mouse support as well.

Extending with custom scripts

However the real power is unleashed when you bind any arbitrary unix commands to the number keys. For example:

mkdir keep maybe discard
feh --scale-down --auto-zoom --recursive --action "mv '%f' discard" --action1 "mv '%f' keep" --action2 "mv '%f' maybe" . &

Here is what is going on in the two commands above. First we create three directories. Next we bring up feh in the directory (.the current directory in this case) where we have copied the images from the memory card and use right left keys to cycle through the images.

The recursive flag takes care of going through any subdirectories. The scale-down and auto-zoom handles the sizing the images properly. The action flag allows you to associate arbitrary unix commands with keys 0-9. And that is incredible!

In the example above hitting the 0 key moves it to the directory discard. This is for two reasons – I am right handed and my workflow is to aggressively discard rather than keep. keep-s are less in numbers and easy to decide, so they are bound to 1. maybe-s are time sinks, so I bind it to 2. I might do a few more passes on each folder before the keep bucket is finalized.

Taking it to the next level

But to take it to the next level, lets bind our 1 (keep) to aws s3 cp command. So we can instantly start uploading them to s3 with one keystroke. Here’s the basic idea:

bucket=`date "+%Y-%m-%d-%s"`
aws s3 mb s3://${bucket}/keep --region us-west-1
aws s3 mv '%f' s3://${bucket}/'%f' &

Note the ampersand at the end of the command – this helps in putting the upload command in the background. That way the upload is not blocking and you can keep going through the images.

This is reasonably stable – even if feh crashes in the middle of your workflow, the upload commands are queued up and continue in the background.

Here is what the final command looks like. You can put this is a script and add it to your path for accessing quickly.

feh --scale-down --auto-zoom --recursive --action "aws s3 mv '%f' s3://${bucket}/'%f' &" --action1 "mv '%f' keep" --action2 "mv '%f' maybe" . &

This workflow is not for those who do a lot of editing with their pictures. Overallfeh is fast to load images and provides a lot of extensibility.

Next Steps

The next step would be to configure the lambda function to the S3 upload event and have the unofficial instagram api post the image to instagram. One step remaining would be including the individual hashtags before S3upload. That way from memory card to instagram can be reduced to just a few keystrokes.

Beyond that, I intend to move feh part of the pipeline to a raspberry pi. I can plug the raspberry pi to the TV of the hotel I am staying at and cut my computer from the loop. Here’s a short dev.to post I wrote up for setting up my raspberry pi with a TV. It will probably be a few weeks to get everything together. Till then enjoy a very reticent feed from my instagram .

[instagram-feed]

Italy

A few pictures from my first Italy trip. It most certainly, would not be the last. The mesmerizing beauty of the land, the layers of history at every corner, the warmth of people and the sumptuous delicacies – makes you fall in love with the country the very moment you lay your first foot in the country. Hope to be back soon!

ChiPy Python Mentorship Dinner March 2015

Chicago Python Users group Mentorship program for 2015 is officially live! It is a three month long program, where we pair up a new Pythonista with an experienced one to help them improve as developers. Encouraged by the success of last year, we decided to do it in a grander scale this time. Last night ChiPy and Computer Futures hosted a dinner for the mentors at Giordano’s Pizzeria to celebrate the kick off – deep dish, Chicago style!

The Match Making:

Thanks to the brilliant work by the mentor and mentees from 2014, we got a massive response as soon as we opened the registration process this year. While the number of mentee applications grew rapidly, we were unable to get enough mentors and had to limit the mentee applications to 30. Of them, 8 were Python beginners, 5 were interested in web development, 13 in Data Science, and rest in Advanced Python. After some interwebs lobbying, some arm twisting mafia tactics, we finally managed to get 19 mentees hooked up with their mentors.

Based on my previous experience at pairing mentor and mentees, the relationship works out only if there is a common theme of interest between the two. To make the matching process easier, I focused on getting a full-text description of their background & end goals as well as their LinkedIn data. From what I heard last night from the mentors, the matches have clicked!

The Mentors’ Dinner:
As ChiPy organizers, we are incredibly grateful to these 19 mentors, who are devoting their time to help the Python community in Chicago. Last night’s dinner was a humble note of thanks to them. Set in the relaxed atmosphere of the pizzeria, stuffed with pizza and beer, it gave us an opportunity to talk and discuss how we can make the process more effective for both the mentor and mentees.

Trading of ideas and skills:
The one-to-one relationship of the mentor and mentee gives the mentee enough comfort for saying – “I don’t get it, please help!”. It takes away the fear of being judged, which is a problem in a traditional classroom type learning. But to be fair to the mentor, it is impossible for him/her to be master of everything Python and beyond. That is why we need to trade ideas and skills. Last time when one of the mentor/mentee pairs needed some help designing an RDBMS schema, one of the other mentors stepped in and helped them complete it much faster. Facilitating such collaboration brings out the best resources in the community. Keeping these in mind we have decided to use ChiPy’s meetup.com discussion threads to keep track of the progress of our mentor and mentee pairs. Here is the first thread introducing what the mentor and mentee are working on.

Some other points that came out of last night’s discussion:

We were not able to find mentors for our Advanced Python track. Based on the feedback we decided to rebrand it to Python Performance Optimization for next time.
Each mentor/mentee pair will be creating their own curriculum. Having a centralized repository of those will make them reusable
Reaching out to Python shops in Chicago for mentors. The benefit of this is far reaching. If a company volunteers their experienced developers as mentors, it could serve like a free apprenticeship program and pave the way in recruiting interns, contractors and full time hires. Hat-tip to Catherine for this idea.

Lastly, I want to thank our sponsor – Computer Futures, for being such a gracious hosts. They are focused on helping Pythonistas find the best Python job that are out there. Thanks for seeing the value in what we are doing and hope we can continue to work together to help the Python community in Chicago.

If you are interested in learning more about being a mentor or a mentee, feel free to reach out to me. Join ChiPy’s meetup.com community to learn more about what’s next for the mentor and mentees.

Chicago Python User Group Mentorship Program

If you stay in Chicago, have some interest in programming – you must have heard about the Chicago Python Users Group or Chipy. Founded by Brian Ray, it is one of the oldest tech group in the city and is a vibrant community that welcomes programmers of all skill levels. We meet on the second Thursday of every month at a new venue with some awesome talks, great food and a lot of enthusiasm about our favorite programming language. Other than talks on various language features and libraries, we have had language shootouts (putting Python on the line with other languages), programming puzzle night etc.

Chipy meetups are great to learn about new things and meet a lot of very smart people. Beginning this October, we are doing a one on one, three month mentorship program. Its completely free, and totally driven by the community. By building this one to one relationships through the mentorship program, we are trying to build a stronger community of Pythonistas in Chicago.

We have kept it open on how the M&M pairs want to interact, but as an overall goal we wanted the mentors to help the mentees with the following:

1. Selection of a list of topics that is doable in this time frame (October 2014 – January 2014)
2. Help the mentee with resources (pair programming, tools, articles, books etc) when they are stuck
3. Encourage the mentee to do more hands on coding and share their work publicly
It has been really amazing to see the level of enthusiasm among the M&M-s. I have been fortunate to play the role of a match maker – where I look into the background, level of expertise, topics of interests and availability for all M&M-s and try to find out an ideal pair. I’ve been collecting data at every juncture so that we can improve the program in later iterations.

Here are some aggregated data points till now:

Signups
# of mentors signed up: 15
# of mentees new to programming: 2
# of mentees new to Python: 16
# of mentee-s Advanced Python: 5
Total: 37

Assignment:
# of mentors with a mentee: 13
# of mentees new to programming with an assigned mentor:1
# of mentees new to Python with an assigned mentor:11
# of mentees with Advanced Python with an assigned mentor:1
Outstanding:
# of mentors for newbie mentees without an assignment: 2
# of mentees unreachable: 4
# of mentees new to programming without an assigned mentor:1 (unreachable)
# of mentees new to Python without an assigned mentor:2 (unreachable)
# of mentees with Advanced Python without an assigned mentor:4 (1 unreachable, 3 no advanced mentors)

Other points:
– Data analysis is the most common area of interest.
– # of female developers: 6
– # of students: 2 (1 high-school, 1 grad student)

All M&M pairs are currently busy figuring out what they want to achieve in the next three months and preparing a schedule. Advanced mentees, are forming a focused hack group to peer coach on advanced topics.
We are incredibly grateful to the mentors for their time and the enthusiasm that the mentees have shown for the program. While this year’s mentoring program is completely full, if you are interested in getting mentored in Python, check back in December. Similarly, if you want to mentor someone with your Python knowledge, please let me know. If you have any tips you would want to share on mentoring, being a smart mentee – please leave them in the comments – I’ll share them with the mentor and mentees. And lastly, please feel free to leave any suggestions on what I can do to make the program beneficial for everyone.

The Third Meetup

Last Tuesday was our third meetup for Chuck Eesley’s venture-lab.org. Instead of the Michigan Street Starbucks opposite to The Chicago Tribune, we pivoted to the Wormhole for this one. For any geek who has been to this place, knows what a riot it is. From “Back to the Future” time-machine retro-fitted on the ceiling, old atari cartridges as showpieces on the coffee-table, super typo-friendly wifi password, stopwatch controlled brewing, Starwars puppets, shopkeep.com app on ipad instead of cashbox – the bearded coffee masters had it all. Everything except a place to accommodate the thunderous 8 of Lake Effect Ventures.

Two hours of caffeine drenched brainstorming spitted out the following:

I sketched out how the process might flow in two steps. We are down to a pretty bare minimum concept build which is ideal both for this class and for getting something up quickly so that we can test it.
I set up a Twitter account for Lake Effect Ventures so that we can tweet about progress we are making.
Andy is going to jot up a positioning statement and beef up the business model canvas for the concept
Leandre will use these to complete our 2-slide initial submission for our deliverable for the next deadline
Leandre will also use this to start to craft a presentation deck
Benn will be working on the copy for the landing page that I started.
Benn will also be crafting a logo in Photoshop (Alex, Zak, Sidi if any one of you is good with design Benn would appreciate the assistance there)
We need to think of a name for the concept as well

We think it is a bit premature to start on the user stories right now given that we have a good idea of what we are gonna build. Charles and me are gonna start on that and look to have something complete from a Version 1.0 standpoint by mid next week barring no setbacks. We will look to craft the user stories once we complete the MVP and use them as structure for testing features and functionality (Zak stay tuned on this)
Benn and Andy will also be working on putting together a more formal customer survey so that we structure the interviews we are having and start to compile meaningful data which we will need going forward.
Its getting exciting ….

Advice:John Doerr on working in teams

Incredible Networking: Collect names, emails of all folks you meet. Be very careful about who your friends and keep in touch – after all you become the average of the five people you spend your time with. Call them up – Its incredible what people will tell you over the phone. (This is something, I have always fallen short – I can hardly get beyond emails).

Carry Chessick, the founder and last CEO of restaurant.com once told me after his lecture session at UIC, that networking as it is perceived is worthless. When you meet people, make sure you finish off by saying “If I can be of any help to you, please do not hesitate to get in touch”. That’s the only way that business card will actually fetch you some benefit. I met a sales guy from SalesForce.com, some time back at Chicago Urban Geeks drink … who sent out a mail immediately after the introduction from his phone with a one line saying who he was, where we met, and that he’ll keep an eye on tech internship notices for me. Brilliant.

360-s: If you want to find information about some company, of course you Google. So lets say if you are gathering info about Google, you’ll also want to talk to their competitors Yahoo, Bing … and find what they are thinking. Then you triangulate all that information to get in a good position.

Coaching: Make sure there is some one will consistently give you advice on what’s going on in your workplace.

Mentoring: Having a very trusted person outside your work who can give advice is invaluable.

Time buddy: How do you make sure that you are doing good time management? Get a time buddy, compare your calendars on how you are spending time. Bill Gates does this Steve Balmer.

Another interesting practice I’ve read sometime back on Hackernews is communicating with team members in two short at regular intervals:
(1) What I did last week/day:
(2) What I’ll do next week/day:

As my dear friend Guru Devanla(https://github.com/gdevanla) would put it “Its all about setting expectations … and meeting them”!

Data loss protection for source code

Scopes of Data loss in SDLC
In a post Wikileaks age the software engineering companies should probably start sniffing their development artifacts to protect the customer’s interest. From requirement analysis document to the source code and beyond, different the software artifacts contain information that the clients will consider sensitive. The traditional development process has multiple points for potential data loss – external testing agencies, other software vendors, consulting agencies etc. Most software companies have security experts and/or business analysts redacting sensitive information from documents written in natural language. Source code is a bit different though.

A lot companies do have people looking into the source code for trademark infringements, copyright statements that do not adhere to established patterns, checking if previous copyright/credits are maintained, when applicable. Blackduck or, Coverity are nice tools to help you with that.

Ambitious goal

I am trying to do a study on data loss protection in source code – sensitive information or and quasi-identifiers that might have seeped into the code in the form of comments, variable names etc. The ambitious goal is detection of such leaks and automatically sanitize (probably replace all is enough) such source code and retain code comprehensibility at the same time.

To formulate a convincing case study with motivating examples I need to mine considerable code base and requirement specifications. But no software company would actually give you access to such artifacts. Moreover (academic) people who would evaluate the study are also expected to be lacking such facilities for reproducibility. So we turn towards Free/Open source softwares. Sourceforge.net, Github, Bitbucket, Google code – huge archives of robust softwares written by sharpest minds all over the globe. However there are two significant issues with using FOSS for such a study.

Sensitive information in FOSS code?

Firstly, what can be confidential in open source code? Majority of FOSS projects develop and thrive outside the corporate firewalls with out the need for hiding anything. So we might be looking for the needle in the wrong haystack. However, being able to define WHAT sensitive information is we can probably get around with it.

There are commercial products like Identity Finder that detect information like Social Security Numbers (SSNs), Credit/Debit Card Information (CCNs), Bank Account Information, any Custom Pattern or Sensitive Data in documents. Some more regex foo or should be good enough for detecting all such stuff …

#/bin/sh
SRC_DIR=$1
for i in `cat sensitive_terms_list.txt`;do
        for j in `ls $SRC_DIR`; do cat $SRC_DIR$j | grep -EHn --color=always $i ; done
done

Documentation in FOSS

Secondly, the ‘release early, release often’ bits of FOSS make a structured software development model somewhat redundant. Who would want to write requirements docs, design docs when you just want to scratch the itch? The nearest in terms of design or, specification documentation would be projects which have adopted the Agile model (or, Scrum, say) of development. In other words, a model that mandates extensive requirements documentation be drawn up in the form of user stories and their ilk. being a trivial example.

Still Looking
What are some of the famous Free/Open Source projects that have considerable documentation closely resembling a traditional development model (or models accepted in closed source development)? I plan to build a catalog of such software projects so that it can serve as a reference for similar work that involve traceability in source code and requirements.

Possible places to look into: (WIP)
* Repositories mentioned above
* ACM/IEEE
* NSA, NASA, CERN

Would sincerely appreciate if you leave your thoughts, comments, poison fangs in the comments section … 🙂