Konduit, Inc: The Perfect Database For Business

Choosing an application database can be challenging. If you are trying to reach a decision, but face multiple issues, know that emphasizing on a few elements can help you make a choice. First and foremost, collect your thoughts on what information you would like to track. Next, decide how long you need to keep data accessible before archiving them. Thereafter, be specific about the features, functions, and automation capabilities  you would want your system to have. Additionally, don’t forget that business needs change over time, so set a budget to include periodic modifications.

If unable to make the right decision, get in touch with an expert. To choose the best online database platform, you can rely on Konduit, Inc., the renowned database applications consulting company. Backed by decades of experience of providing structured business applications in an affordable recurring pricing model, Konduit, Inc., is a celebrated company that individuals prefer for the database development of their business. Konduit makes use of Transaction Editor {software as a Service (SaaS), is for small businesses or departments} for database development so as to gain maximum flexibility. Also, Konduit’s database development uses time tested normalization principles that speed-up long range queries.

So, if you want your business database development done right, go nowhere, but Konduit, Inc.

To know more, check the Konduit site

What is a database service?

Database as a Service (DBaaS) may sound like a service business, but really its just another specialized Infrastructure as a Service (IaaS). The CTO Vision article on DBaaS talks about hosting a database management system, but not hosting the application itself. Specifically, the author states “I believe that the reason for the draw is that databases, while critical to your applications, are generally a bit difficult to configure, scale and operate correctly. When presented with the opportunity to off-load that work to a provider, developers jump at the chance.” Developers seek help with databases because they were taught little in school about them and their experience becomes clouded by the many misperceptions about relational databases best described by Fabian Pascal. A good starter article is “Data Management’s Misconceptions” by Joab Jackson.

Software application development that relies on mash-ups of packaged products without professional services to clarify and organize the meaning of data is, well, meaning-LESS. Database management systems and programming languages cannot, by themselves, provide the meaning, integrity, and consistency that gives an application purpose and usefulness. Some ONE needs to be facilitating the discussion of what functionality is needed, and to clarify what the nouns and verbs used to describe a business process mean. Without this clarity of purpose, any software application development will eventually be doomed.

Konduit combines the database development and application development services needed to accelerate the iterative cycle of process adaptation to empower young businesses with a virtual “IT Department”. This service is delivered over an encrypted connection on the Internet, requires little upfront cost, no contracts, and a predictable recurring monthly fee structured to reflect the complexity of the application delivered.  But more importantly, this service includes all the database development, application development, workflow design, hosting services, data stewardship, and end-user support.

The term “database” is filled with mystery, but in reality it’s just about communication and logical organization, something every business needs in order to thrive. The more organized your business information is, the greater chance you will have to settle misunderstandings with your customers and vendors. If you can audit the history of data changes, you can identify and remedy mistakes and know who needs more training.  Database management systems provide a critical part of what is needed.  Information technology is more than hardware and software as commodities. Information technology REQUIRES PEOPLE who understand your business and can rapidly adapt your software-based processes to changes in the marketplace or operations.  To be effective, a freelance web programmer must understand how to use a database management system in the context of a business process automation application.

Database Design is Easy — Or Is It?

I came across a Request for Proposal from a government entity whose sole purpose was to conform to the data collection standards of a superior government entity.   The following entry is paraphrased for brevity.

Yes/No Questions. The “yes” response and the “no” response each has its own column in the spreadsheet tool.   To specify “yes” for a given question, the user should enter a “Y” in the given question’s “yes” column. To specify “no” for a given question, the user should enter an “N” in the given question’s “no” column.  One question on the form is an exception to this “standard” where there is only one column in the spreadsheet tool for this question, and users should enter a “Y” for “yes”  and “N” for “no”.

This was written by professionals who should be in a different profession.  I struggle with how to explain why this is fubar.  So here goes.

First of all, these are questions on a form, questions, mind you, with only two answers, “yes” or “no”.  Really.  How complicated can that be?  Apparently, very complicated.  If two responses (one in the Yes column and the other in the No column) are allowed, how will one know which response reflects the facts of the respondent?

The noted “exception” should actually be the “standard”.   Most likely, this was written by lawyers to allow for flexible interpretation of the law when enforcing compliance so as not to hurt anyone’s feelings.  Or maybe it exists to confuse the respondent, so they will give up filling out the form entirely to seek out other sources of assistance.  Whatever the reason, the person designing such forms should be in a different line of work because either their brain or their heart are not in it for the right reason.

Why is this important?  Say you wanted to report on the number of respondents who replied to a particular question.  Count the ones who respond “yes”.  Count the ones who respond “no”.  Simple, right?  OK.   Now calculate the percentage of respondents for each.  Well, if some put down “Y” in the Yes column AND “N” in the No column, the sum of the counts of each would be more than the total number of respondents.

Now, image you are a legislative leader and you are reading a compliance report based on these statistics in order to place a vote for a multi-billion dollar extension of the program.  You might see that 65% said “Yes” and 63% said “No” to a particular question on which you would like to base your decision.   Or maybe, the data analyst preparing the “required pie chart” report will recognize the data problem, and fudge the responses so the chart reflects HIS political preferences.   Democracy in action.

I have all the answers, its knowing the right the questions to ask that is the challenge.

For more examples of database development challenges, please visit my favorite website, www.dbdebunk.com.  These design problems are not limited to government entities.  There are plenty of examples in commercial enterprises.

Freelance Web Programmer for Small Business

Marketing your services as a freelance web programmer is a tough gig.  But doing so in the United States Software Development Market has become increasingly more challenging.  Optimizing your website for search engines, advertising through Google or Facebook, and sending countless postcards and emails are just no match for the well connected and HR-focused efforts of recruiters.  The old adage “it’s who you know rather than what you know” that drives your success is still true.  But as a provider of application development services, getting you to know me is more than half the battle.

Technology recruiters are extensions of large corporate HR departments.  They work their network of buyers and sellers to earn their healthy markups.  HR Staffers specialize in labor law.  Technology recruiters find candidates with specialized skills.  The problem is, if you are a small business owner without a techno-priest advocating for a specific set of skills, you are left with thousands of technology choices and nowhere to turn for a solution to workflow design and application development.  Oftentimes, a power user or student will have provided a “temporary solution” in Excel or Access that is difficult to support as the business requirements change.

I’ve worked for plenty of recruiters, but recent experiences have demonstrated that disintermediation by web-based freelancer project posting sites are attempting to take market share from sites like Dice.com and Guru.com, who have been around much longer and are more supportive of recruiters.  Some are trying to replace recruiters entirely by claiming to have vetted their applicant’s skill sets.  Unfortunately for them,  recruiting is a local business founded on years of developed relationships.

I don’t know if recruiters ever get calls from small business owners looking for help.  But if they do, I expect that their lower budgets and inability to sign for a minimum engagement of three or more months would prevent a deal.  So the small business owner has to ask a friend, their desktop support provider, their PC/Printer/networking retailer, the local Small Business Association chapter, or local technical college (collectively “small business associates”) for a referral.

What if any of these small business associates were able to refer someone who can help them with a small project and is available within three times zones, can provide continuous support, and costs a mere fraction of the cost of hiring a full or part time application developer?  That is what Konduit does. And we are willing to share with small business associates some of this continuous micro-revenue stream for simple referrals that result in new subscribers.

I have provided on my website TransactionEditor.com a random display of the descriptions of projects we are bidding on.  You can find this at the bottom of the “Portfolio-Your Application” page, under the heading “Posted Projects (RFQ,RFP) Suitable for TEd“. This helps our referrers understand the scope we are focused on.  This should also help small business owners determine whether we might be able to help them.  Along with the description, I rate the quality of the mostly unaltered description as Good, Better, Best:

  • Good = Vague description, but function-related rather than skillset-related.
  • Better = Majority of functionality within TEd capability, but some items require clarification.
  • Best = Well documented functional description of requirements.

With over 20 years of experience, my services also include database development, application development, and workflow design.

Computer Science Degree — Is It Worth It?

Here is a request for a proposal taken from a popular freelance-web-programmer site (spelling errors included):

I am looking for someone who has good experience writing thesis projects. I have a deadline of December 10th to submit my thesis. I am a Computer Science student and almost at the end of my completion of the degree. I have to present a good strong thesis project related to IT which has some programming or implementation involved in it. Below is the brief description which my professor has given me to write my thesis. I am looking for someone who should be good at writing strong tehnical thesis. Money woudl not be an issue for the right person.

I come across these occasionally and wonder what will this person do when they enter the “real world”.  This one is particularly telling with its directness.  Will an employer be able to detect their lack of ability?  Will they be fast-tracked to management because of their pedigree?  Will their spell-checker prevent future exposure to their incompetence?  Or will they eventually resign in shame like the CIO of Equifax?

I believe, for the most part,  there are good schools producing good students.  But seeing things like this worry me.  The general lack of enthusiasm of Millennials and the misplaced emphasis on job placement among colleges (and universities) has degraded our educational system.  And, assuming this is a foreign college, I have much doubt about our ability to fulfill the vacant computer programming positions by increasing the H1B visa quotas.

 

Why Your Data Loads Take Too Long

During twenty years of doing application development for clients, I have optimized process automation data loads primarily by changing INSERT statements into a BULK LOAD process.  In SQL Server, an external command line utility (bcp.exe) allows bidirectional data streams to flow.  In MySQL, the inward flow is managed by the LOAD DATA INFILE command.

The key to using these BULK LOAD processes is formatting the data properly in files before loading the data.  The best language for parsing large streams of data is Perl.  If you convert your INSERT  statement-generating PHP, Java, C#, VB, etc. to a Perl scrubbing and formatting operation and use the database’s BULK LOAD utility, I am sure you will see a significant increase in the performance of your data loads.  I have used operating system shell scripting languages to tie it all together for automation.

Another factor is the configuration settings of the relational database logging operation.  Most simplified databases do not log transactions because they do not enforce ACID principals.  In fact, most databases, relational or not, are based on straight-forward ISAM principals.  Transactional databases, those that require recovery of transactional (e.g. INSERT) failures, perform extra operations to allow for that recovery.   But what most freelance web programmers don’t know is that this extra activity at every INSERT of a row slows the load process down significantly.  In a data warehouse, this transactional logging can be turned off.  Data recovery is controlled by managing the files used during the bulk load operation.

A BULK LOAD operation in transactional databases will treat the entire load as a single transaction, so breaking up a large set of rows into separate files may make the end-user support more manageable.  The cause of failed file loads can be investigated and dealt with on a more granular level if the files are split into manageable pieces.  What defines “manageable” is determined on a case by case basis by the latency and use requirements of the data.

 

 

Platform (PaaS) , not Infrastruture (IaaS), as a Service

Jonathon Feldman wrote in February “Why IaaS Won’t Happen in Most Enterprises“.   He makes the argument that departmental business needs will drive spending on Platform as a Service provisioning.   The argument is a solid one that Konduit has been using to satisfy end-user needs for over ten years.

Like software application development tool vendors, most PaaS providers rely upon a following of freelance web programmers to train and certify in their technology, all at significant time and dollar investment.  If they attract enough followers, then major tool vendors eye them for acquisition, which is followed by migration or deprecation, leaving customers to start over again.

When PaaS licensing is combined with application development services so that customers are not left with unsupported mash ups, we call that Support.  The development, deployment, and support of applications written using Konduit’s Transaction Editor Platform as a Service stays within the control of Konduit engineers.  A customer’s subject matter experts remain in control of the work flow and practices that define the software’s functionality, without having to understand the mechanics underlying the computer logic, security, reliability, or presentation methodologies.  They just get to use it and share it as needed.  In essence, Support as a Service, provides infrastructure, platform, and services to the end-user without requiring their understanding of the technology.  Unfortunately the acronym SaaS has already been taken.  Please comment with your ideas for a new acronym for Support as a Service.

 

Cloud Economies of Scale – Not!

Over the past several years, software vendors have been moving their customer-premise offerings to cloud services.  The cost of software license fees are added to costs for hardware, network, and security infrastructure.  Millions of dollars are being invested in putting packaged software in the cloud so customers can expense rather than amortize their information technology purchases.  Like the record industry, vendors create bundles (i.e. albums) in order to stack less-used features with their core features.  Monthly subscription service rates are calculated like lease agreements, so accountants can depreciate these investments.  The assumption is that economies of scale apply to technology.

These investments ignore the fact that technology is distributing workloads, not consolidating them. The Internet of Things is adding points of input.  Arrays of commodity smartphone components continue to replace multi-million dollar engineered products from digital microscopes  to satellites and cable TV.  The Internet is still made up of mostly personal computer technology all wired together.  Blade server technology uses a distributed design to imbue redundancy, and thereby reliability, into their products.  Distributed computing is the only possible way to solve the most difficult problems.   Unfortunately, software design has not kept pace.

The act of creating “packaged” or “common off-the-shelf” (aka “COTS”) software products requires accommodating the needs of a diverse set of users.  This results in a small (20%) subset of features that remain useful for any particular group of users, while the majority of features (80%) remain unused. Custom software, by contrast,  contains only those features that users deem useful.

The development and support of custom application development requires knowledge of how business processes operate.  Small enterprises that cannot afford a full-time programmer or have difficulty retaining staff due to volatile business cycles, need a process automation solution that applies common computing principals while remaining flexible in the data elements it can handle.  Spending millions of dollars customizing an enterprise package is out of reach for most small enterprises.  (Large enterprises just waste a lot of time pretending to use more features.)

Most small enterprises rely upon spreadsheets and word processors to digitize their paper document world.  Vendors reply with file-based solutions to help them “organize” their digitized documents.  Few vendors offer structured data solutions (i.e. database development) that enable searching, sorting, and sharing of the transactional details of documents.  Companies that combine the ease of use of a spreadsheet with the accountability of a database using a common business process methodology can solve this problem.   A common business process methodology enables Konduit.com to highly customize application development for each client, while retaining a secure, cost-effective operational environment.

Scalability of the cloud is not achieved by moving bloated software to mainframe hardware.  Cloud scalability is achieved by distributing only “needed” features and applying a common business process to a unique set of data items. This allows software and hardware to be shared while content remains specific and relevant.

Transactions and References

As I speak with people from various technical backgrounds, I find there is lot of confusion about how to structure data.  For a full background on this and related topics I rely upon the work of Fabian Pascal, C.J. Date, and C.F. Codd, vibrantly maintained on Pascal’s site,  dbdebunk.com.  I encourage anyone who favors “learning” over “training” to spend some time there.

When I address a new challenge of database development or workflow design, I start by distinguishing between Transaction data and Reference data.  Transaction data are those that accumulate over time, but do not change frequently.  There are a lot of INSERTs, but not many UPDATEs.  Orders, invoices, calls, observations, and events are all types of Transaction data.  Transaction data may include attributes that contain Reference data.

Reference data are those that remain mostly static over time, but once established, change infrequently.  Examples include addresses, phone numbers, product lists, clients, vendors, etc.  Reference data grows at a much slower pace than Transaction data, so much of the maintenance involves de-duplication and consistency of form.  For instance, address data can be better de-duplicated if at first you make all the street types consistent: i.e. “Avenue” and “Ave” and “Av” are all made to be the same “Ave”.  If you can do this BEFORE inserting it into a database, that saves A LOT of time and resources.

By contrast, Transaction data maintenance is more focused on removing data that are no longer needed.  The most efficient way to manage Transaction data in a database is to 1) not load attributes that will never be used, and 2) partition data based on attributes that reflect its useful life.  This makes it easier to remove it without scanning all the data you need to keep.

So the next time you are thinking about loading up your linked lists, name-value pairs, and dictionaries into a database, remember that you only need to save what you will use.  Kind of like your refrigerator.  If you are not going to eat it, you don’t need to wrap it up or freeze it.  Collect only what you need.   Determining what you “need” is of course a prerequisite.

I Want to Own my Own Software — Really?

Here is a plea from a small business owner who wants to own their own software:

“I had a programmer, who is no longer in the picture, develop software for me.  He placed the program on his server. I can use the program but, I cannot register others to use it, etc. I need someone to put the program on MY server where I have control.”

OK.  So you own the intellectual property rights to some code that helps your business succeed.  But what if your business needs change?  You still need someone to take responsibility for understanding that code, have the skills to modify that code, and be interested and compensated sufficiently to continue to support that code.   If you own it, now you are in the software business.  You need to keep an employee or a freelance web programmer interested in your business application long enough to survive the next operating system or framework upgrade, business process change, or hardware upgrade.

What if the framework vendor decides it no longer wants to support that framework?  Vendor acquisitions, product sunsets, or management changes usually result in someone (or a lot of someones) rewriting the same code over again every 3-5 years.   So now your precious intellectual property is made obsolete by forces beyond your control, those external forces operating behind the scenes that only seasoned IT executives understand all too well.

But you just want to run your business as efficiently as possible.  A software package made for your industry seems appropriate.     That is, until that vendor gets gobbled up by a competitor or makes upgrades to features you don’t use.  Again, you are back to  rebuilding into code your unique business process–your trade secrets–that make your business run efficiently.

Standards.  Yes, the answer is standards.  But what is the standard today?  When and how will that standard change over time, and will that affect your application development? This is a moving target that is at the bleeding edge of the software field.  For business applications that process transactions, does this really need to change as often as software vendors churn out new features?  Why should your IT support persons change more often than your accountant?

Business computer systems serve two purposes.  They keep track of Things and they keep track of Events (i.e. transactions).  For example: “Sold 34 Widgets on Thursday”.  “Widgets”=Thing.  “Sold 34 on Thursday”=Event.  Not a lot has changed in 40 years with respect to the technology needed to handle this.  HTML behaves like CICS.  Relational databases replaced ISAM.  The software industry’s penchant for “new and improved” is why spreadsheets run half the world.  But I digress.

If you focus on the SERVICES related to keeping order among your chaos, you will likely succeed with less technology pain.  Like having a good CPA keep track of your books or an ATTORNEY keep track of your contracts, having a good professional keep track of your software development will likely make you more successful, measured in your preference of either more profit or more free time.

So what is this type of “professional” called?  There isn’t one name for such a person.  There are no well known licensing boards certifying information technology professionals.  In fact, if you asked a hiring manager (or professional recruiter for that matter) what to look for in a freelance web programmer, they would list a litany of products, technologies, and programming languages.  If carpenters were hired like programmers, the interview for a “Walnut Guru” would go something like this.  If you read that link, you get my point.

Bottom line, technology matters, but only if you have the professionals who can sift through the pile and find your pony in there.