Computer Science Degree — Is It Worth It?

Here is a request for a proposal taken from a popular freelance-web-programmer site (spelling errors included):

I am looking for someone who has good experience writing thesis projects. I have a deadline of December 10th to submit my thesis. I am a Computer Science student and almost at the end of my completion of the degree. I have to present a good strong thesis project related to IT which has some programming or implementation involved in it. Below is the brief description which my professor has given me to write my thesis. I am looking for someone who should be good at writing strong tehnical thesis. Money woudl not be an issue for the right person.

I come across these occasionally and wonder what will this person do when they enter the “real world”.  This one is particularly telling with its directness.  Will an employer be able to detect their lack of ability?  Will they be fast-tracked to management because of their pedigree?  Will their spell-checker prevent future exposure to their incompetence?  Or will they eventually resign in shame like the CIO of Equifax?

I believe, for the most part,  there are good schools producing good students.  But seeing things like this worry me.  The general lack of enthusiasm of Millennials and the misplaced emphasis on job placement among colleges (and universities) has degraded our educational system.  And, assuming this is a foreign college, I have much doubt about our ability to fulfill the vacant computer programming positions by increasing the H1B visa quotas.

 

Why Your Data Loads Take Too Long

During twenty years of doing application development for clients, I have optimized process automation data loads primarily by changing INSERT statements into a BULK LOAD process.  In SQL Server, an external command line utility (bcp.exe) allows bidirectional data streams to flow.  In MySQL, the inward flow is managed by the LOAD DATA INFILE command.

The key to using these BULK LOAD processes is formatting the data properly in files before loading the data.  The best language for parsing large streams of data is Perl.  If you convert your INSERT  statement-generating PHP, Java, C#, VB, etc. to a Perl scrubbing and formatting operation and use the database’s BULK LOAD utility, I am sure you will see a significant increase in the performance of your data loads.  I have used operating system shell scripting languages to tie it all together for automation.

Another factor is the configuration settings of the relational database logging operation.  Most simplified databases do not log transactions because they do not enforce ACID principals.  In fact, most databases, relational or not, are based on straight-forward ISAM principals.  Transactional databases, those that require recovery of transactional (e.g. INSERT) failures, perform extra operations to allow for that recovery.   But what most freelance web programmers don’t know is that this extra activity at every INSERT of a row slows the load process down significantly.  In a data warehouse, this transactional logging can be turned off.  Data recovery is controlled by managing the files used during for bulk load operation.

A BULK LOAD operation in transactional databases will treat the entire load as a single transaction, so breaking up a large set of rows into separate files may make the end-user support more manageable.  The cause of failed file loads can be investigated and dealt with on a more granular level if the files are split into manageable pieces.  What defines “manageable” is determined on a case by case basis by the latency and use requirements of the data.

 

 

Platform (PaaS) , not Infrastruture (IaaS), as a Service

Jonathon Feldman wrote in February “Why IaaS Won’t Happen in Most Enterprises“.   He makes the argument that departmental business needs will drive spending on Platform as a Service provisioning.   The argument is a solid one that Konduit has been using to satisfy end-user needs for over ten years.

Like software application development tool vendors, most PaaS providers rely upon a following of freelance web programmers to train and certify in their technology, all at significant time and dollar investment.  If they attract enough followers, then major tool vendors eye them for acquisition, which is followed by migration or deprecation, leaving customers to start over again.

When PaaS licensing is combined with application development services so that customers are not left with unsupported mash ups, we call that Support.  The development, deployment, and support of applications written using Konduit’s Transaction Editor Platform as a Service stays within the control of Konduit engineers.  A customer’s subject matter experts remain in control of the work flow and practices that define the software’s functionality, without having to understand the mechanics underlying the computer logic, security, reliability, or presentation methodologies.  They just get to use it and share it as needed.  In essence, Support as a Service, provides infrastructure, platform, and services to the end-user without requiring their understanding of the technology.  Unfortunately the acronym SaaS has already been taken.  Please comment with your ideas for a new acronym for Support as a Service.

 

Choosing The Right Inventory Control Software For Small Business

Complete control over your inventory is one of the most crucial aspects of enjoying long-term success. When you have one or more warehouses to manage, making sure that they are properly stocked, orders are properly taken and delivered, and all changes properly reported can make or break your business.

Small business inventory control has its pitfalls. As a small business yourself, you could experience these pitfalls while choosing an inventory control software for small business that has the following issues – it is too costly to maintain, it doesn’t come with the flexibility of being modified easily as your business needs change, it doesn’t adapt to your needs – doesn’t control/print/manage sales orders, shipping advice notices, invoices, and warehouse releases among other things.

On the other hand, if you have the right inventory control/management system in place, you will not only be able to avoid the troubles associated with repetitive inventory tasks, but also replenish your stock in the best possible manner, at the right time.

You don’t need to be an expert to choose the right inventory solution for your small business inventory needs. You need a software that can perfectly meet your needs – streamline warehouse management without needing additional resources or expenses.

Cloud Economies of Scale – Not!

Over the past several years, software vendors have been moving their customer-premise offerings to cloud services.  The cost of software license fees are added to costs for hardware, network, and security infrastructure.  Millions of dollars are being invested in putting packaged software in the cloud so customers can expense rather than amortize their information technology purchases.  Like the record industry, vendors create bundles (i.e. albums) in order to stack less-used features with their core features.  Monthly subscription service rates are calculated like lease agreements, so accountants can depreciate these investments.  The assumption is that economies of scale apply to technology.

These investments ignore the fact that technology is distributing workloads, not consolidating them. The Internet of Things is adding points of input.  Arrays of commodity smartphone components continue to replace multi-million dollar engineered products from digital microscopes  to satellites and cable TV.  The Internet is still made up of mostly personal computer technology all wired together.  Blade server technology uses a distributed design to imbue redundancy, and thereby reliability, into their products.  Distributed computing is the only possible way to solve the most difficult problems.   Unfortunately, software design has not kept pace.

The act of creating “packaged” or “common off-the-shelf” (aka “COTS”) software products requires accommodating the needs of a diverse set of users.  This results in a small (20%) subset of features that remain useful for any particular group of users, while the majority of features (80%) remain unused. Custom software, by contrast,  contains only those features that users deem useful.

The development and support of custom application development requires knowledge of how business processes operate.  Small enterprises that cannot afford a full-time programmer or have difficulty retaining staff due to volatile business cycles, need a process automation solution that applies common computing principals while remaining flexible in the data elements it can handle.  Spending millions of dollars customizing an enterprise package is out of reach for most small enterprises.  (Large enterprises just waste a lot of time pretending to use more features.)

Most small enterprises rely upon spreadsheets and word processors to digitize their paper document world.  Vendors reply with file-based solutions to help them “organize” their digitized documents.  Few vendors offer structured data solutions (i.e. database development) that enable searching, sorting, and sharing of the transactional details of documents.  Companies that combine the ease of use of a spreadsheet with the accountability of a database using a common business process methodology can solve this problem.   A common business process methodology enables Konduit.com to highly customize application development for each client, while retaining a secure, cost-effective operational environment.

Scalability of the cloud is not achieved by moving bloated software to mainframe hardware.  Cloud scalability is achieved by distributing only “needed” features and applying a common business process to a unique set of data items. This allows software and hardware to be shared while content remains specific and relevant.

Transactions and References

As I speak with people from various technical backgrounds, I find there is lot of confusion about how to structure data.  For a full background on this and related topics I rely upon the work of Fabian Pascal, C.J. Date, and C.F. Codd, vibrantly maintained on Pascal’s site,  dbdebunk.com.  I encourage anyone who favors “learning” over “training” to spend some time there.

When I address a new challenge of database development or workflow design, I start by distinguishing between Transaction data and Reference data.  Transaction data are those that accumulate over time, but do not change frequently.  There are a lot of INSERTs, but not many UPDATEs.  Orders, invoices, calls, observations, and events are all types of Transaction data.  Transaction data may include attributes that contain Reference data.

Reference data are those that remain mostly static over time, but once established, change infrequently.  Examples include addresses, phone numbers, product lists, clients, vendors, etc.  Reference data grows at a much slower pace than Transaction data, so much of the maintenance involves de-duplication and consistency of form.  For instance, address data can be better de-duplicated if at first you make all the street types consistent: i.e. “Avenue” and “Ave” and “Av” are all made to be the same “Ave”.  If you can do this BEFORE inserting it into a database, that saves A LOT of time and resources.

By contrast, Transaction data maintenance is more focused on removing data that are no longer needed.  The most efficient way to manage Transaction data in a database is to 1) not load attributes that will never be used, and 2) partition data based on attributes that reflect its useful life.  This makes it easier to remove it without scanning all the data you need to keep.

So the next time you are thinking about loading up your linked lists, name-value pairs, and dictionaries into a database, remember that you only need to save what you will use.  Kind of like your refrigerator.  If you are not going to eat it, you don’t need to wrap it up or freeze it.  Collect only what you need.   Determining what you “need” is of course a prerequisite.

Database Design is Easy — Or Is It?

I came across a Request for Proposal from a government entity whose sole purpose was to conform to the data collection standards of a superior government entity.   The following entry is paraphrased for brevity.

Yes/No Questions. The “yes” response and the “no” response each has its own column in the spreadsheet tool.   To specify “yes” for a given question, the user should enter a “Y” in the given question’s “yes” column. To specify “no” for a given question, the user should enter an “N” in the given question’s “no” column.  One question on the form is an exception to this “standard” where there is only one column in the spreadsheet tool for this question, and users should enter a “Y” for “yes”  and “N” for “no”.

This was written by professionals who should be in a different profession.  I struggle with how to explain why this is fubar.  So here goes.

First of all, these are questions on a form, questions, mind you, with only two answers, “yes” or “no”.  Really.  How complicated can that be?  Apparently, very complicated.  If two responses (one in the Yes column and the other in the No column) are allowed, how will one know which response reflects the facts of the respondent?

The noted “exception” should actually be the “standard”.   Most likely, this was written by lawyers to allow for flexible interpretation of the law when enforcing compliance so as not to hurt anyone’s feelings.  Or maybe it exists to confuse the respondent, so they will give up filling out the form entirely to seek out other sources of assistance.  Whatever the reason, the person designing such forms should be in a different line of work because either their brain or their heart are not in it for the right reason.

Why is this important?  Say you wanted to report on the number of respondents who replied to a particular question.  Count the ones who respond “yes”.  Count the ones who respond “no”.  Simple, right?  OK.   Now calculate the percentage of respondents for each.  Well, if some put down “Y” in the Yes column AND “N” in the No column, the sum of the counts of each would be more than the total number of respondents.

Now, image you are a legislative leader and you are reading a compliance report based on these statistics in order to place a vote for a multi-billion dollar extension of the program.  You might see that 65% said “Yes” and 63% said “No” to a particular question on which you would like to base your decision.   Or maybe, the data analyst preparing the “required pie chart” report will recognize the data problem, and fudge the responses so the chart reflects HIS political preferences.   Democracy in action.

I have all the answers, its knowing the right the questions to ask that is the challenge.

For more examples of this kind of thing, please visit my favorite website, www.dbdebunk.com.

I Want to Own my Own Software — Really?

Here is a plea from a small business owner who wants to own their own software:

“I had a programmer, who is no longer in the picture, develop software for me.  He placed the program on his server. I can use the program but, I cannot register others to use it, etc. I need someone to put the program on MY server where I have control.”

OK.  So you own the intellectual property rights to some code that helps your business succeed.  But what if your business needs change?  You still need someone to take responsibility for understanding that code, have the skills to modify that code, and be interested and compensated sufficiently to continue to support that code.   If you own it, now you are in the software business.  You need to keep an employee or a freelance web programmer interested in your business application long enough to survive the next operating system or framework upgrade, business process change, or hardware upgrade.

What if the framework vendor decides it no longer wants to support that framework?  Vendor acquisitions, product sunsets, or management changes usually result in someone (or a lot of someones) rewriting the same code over again every 3-5 years.   So now your precious intellectual property is made obsolete by forces beyond your control, those external forces operating behind the scenes that only seasoned IT executives understand all too well.

But you just want to run your business as efficiently as possible.  A software package made for your industry seems appropriate.     That is, until that vendor gets gobbled up by a competitor or makes upgrades to features you don’t use.  Again, you are back to  rebuilding into code your unique business process–your trade secrets–that make your business run efficiently.

Standards.  Yes, the answer is standards.  But what is the standard today?  When and how will that standard change over time, and will that affect your application development? This is a moving target that is at the bleeding edge of the software field.  For business applications that process transactions, does this really need to change as often as software vendors churn out new features?  Why should your IT support persons change more often than your accountant?

Business computer systems serve two purposes.  They keep track of Things and they keep track of Events (i.e. transactions).  For example: “Sold 34 Widgets on Thursday”.  “Widgets”=Thing.  “Sold 34 on Thursday”=Event.  Not a lot has changed in 40 years with respect to the technology needed to handle this.  HTML behaves like CICS.  Relational databases replaced ISAM.  The software industry’s penchant for “new and improved” is why spreadsheets run half the world.  But I digress.

If you focus on the SERVICES related to keeping order among your chaos, you will likely succeed with less technology pain.  Like having a good CPA keep track of your books or an ATTORNEY keep track of your contracts, having a good professional keep track of your software development will likely make you more successful, measured in your preference of either more profit or more free time.

So what is this type of “professional” called?  There isn’t one name for such a person.  There are no well known licensing boards certifying information technology professionals.  In fact, if you asked a hiring manager (or professional recruiter for that matter) what to look for in an freelance web programmer, they would list a litany of products, technologies, and programming languages.  If carpenters were hired like programmers, the interview would go something like this.  If you read that link, you get my point.

Bottom line, technology matters, but only if you have the professionals who can sift through the pile and find your pony in there.

 

What is a Software Application?

In 1983, as an analyst for a respected information technology industry marketing research organization1, I once had a debate with another analyst on how to classify a database management system.  Was it “application software” or an “operating system”?

I argued that applications, or at least the processing logic of software applications, were being built in the database language, SQL, and this made it part of the application software arena.  My counterpart argued that database management systems controlled access to the disk hardware to the extent2 that no file system was required.  I said that the “file system” was separate from the “operating system”3, could possibly be considered “application software” by itself, and so on, and so forth.  I lost that argument in 1983, but I think today an analyst would be hard-pressed to succeed in convincing anyone that a database was an operating system.  In fact, today’s virtual software platforms turn present day “operating systems” into “application software”.

IBM’s VM operating system was just beginning to take shape in 1983.  Today, most companies are relying on VMWare to run their businesses and not looking back.  Desktop operating systems are even being “deployed” via VMWare to terminal emulators, now called Internet browsers.  How the evolution of CISI became Chrome today was a world-wide cooperative effort.

But is the Internet browser dead? Apple’s venture into the touch-screen user interface powered by client-based code, and Google’s foray into the same architecture with Android, has brought the “fat client” back.  With Apple it worked because they controlled the hardware and limited the user interfaces to a small number of dimensions.  Android has since blown that away in market share and a plethora of screen dimension choices and accessory options that create insurmountable cross-platform testing challenges.

There is a New Hope.  After ten years of existence, HTML5 and CSS3 are finally getting some attention. These two new standards bring back the Internet browser to the application software domain because they support a wide variety of user-interface controls, while providing greater control over adaptation to displays.  This forethought occurred before the invention of the iPhone because such is the evolution of software.

To stay ahead of these changes, reduce your ever-redundant software development efforts, and truly empower your staff to seek out new applications, to boldly solve the backlog of end-user demands, and breath new life into your business processes, you need to rethink the Internet browser as the primary user interface for your applications.

Konduit provides custom application development and workflow design delivered over the Internet that combines all IT services: database development, hosting services, data stewardship, and end-user support into a single plan.

Footnotes:
1 INPUT (acquired in 2010 by Deltek, Inc.), was bigger than Gartner Group at the time.
2 Pun intended
3 The file system is an API to the disk like shells are user interfaces.

There is Nothing "Standard" About Software

The Holy Grail of application development is a tool that empowers the individual to “build it themselves.”  Spreadsheets, WordPress, and SharePoint seem to lead in that space.  There is no shortage of application development tools.   You can search the Internet for application development tools only to discover that most solutions are so focused and simplistic, they hardly do anything.  You can spend six months to learn how to build your application, only to spend the next six months upgrading it for the next release, until the vendor “sunsets” your platform to make way for the Next Big Thing.

To make software useful requires people who understand your business.  Unfortunately, staffing or finding a freelance web programmer who knows the particular language, libraries, or tools that your application is built upon is no small task.  This is made even more challenging by the ever-improving offers by old and new vendors.  Technical recruiters and HR people  struggle with defining the skills needed because there are literally hundreds of ways to build a business software application.  There really are no “standards”.

What businesses are lacking is standards, real standards.  Imagine what software industry “standards” would read like if they were applied to the construction industry.

Pipe Fittings.  Available in a variety of materials, you can connect your toilet to your water source by configuring your tank prefilter with any number of other Open Source materials.  After extruding the material through your 3D printer and threading the connectors with just about any threading tool, you can form the outer casing to fit your team’s own wrench set.  For a fee, we can provide you with our wrench set.

Wall Components.    We give you everything you need to build your own house.   Solid Pine logs in standard lengths can be trimmed to any dimension using freely available Open Source band saws.  They can easily be bound together with your choice of off-the-shelf components including rope,  tar, steel shanks (headed or headless), and leather trimmings.

Electrical Wiring.   Our professional installers can get you up and running in no time.  We work in one metal (copper, aluminum, brass), use only certified Acme parts, and will give you copyright and patents on every bend and connection made throughout your house.

In my article “I Want To Own My Own Software — Really?“, I reference an mock interview dialog called “If Carpenters Were Hired Like Programmers“.