Why is it so hard to hire software engineers?

As an Engineering Manager at one of my previous employs, we would submit postings looking for something called a 'full-stack employee'. Ideally, this would be someone with expertise in exactly the stack that we had, who has worked in the entire stack before, enough to get proficiency at every single level. For a Microsoft stack, this would be something like .NET, SQL Server, Windows 10, Linux = LAMP (showing my age a bit), or more recently MAMP.

On it's face, it seems impossible. It's highly unlikely that we'd ever find someone, aside from a previous employee, perhaps, who matches technology-for-technology. I've given and received many interviews and can safely say that this requirement is one of the first things to go (unless its heavily prescribed from the top-down, which can be a bit of a problem). More recent coding interview trends use algorithmic knowledge as a metric, or solving pet problems in real time (which I argue is probably less a test of individual programming skill, and more a test of ability to memorize and dedication to training).

If we do it by the numbers, let's say n is the number of layers in your stack. Then 2^n is the combination of potential experiences with different technologies of your stack (assuming 1 technology per layer - an invalid assumption). You want exactly 1 combination of these 2^n possibilities when you want a 'full-stack developer' in your stack. Traditional 3-tier architecture means 1:8 chance of meeting your requirement. Add to that stacks, queues, message queues, batches and so on, and it's not unreasonable to get to the 1:32767 (1:2^7 - and again showing my age - can you guess how?).

If we loosen up on the specific technology requirement (and say, want someone who has *any* experience building an API, UX, and DB), then we can stay at 1:8. The numbers don't lie - you could assume that 1:8 candidates is what you're looking for here too. What I've seen lately is a spike in the number of UX developers, and a dwindling in the number of API developers. That's driven me to the totally-anecdotal analysis that API developers are harder to find than UX developers generally. Just a thought.

Suppose your candidate satisfies that test. You still don't know if the individual can code, or considers system design when they do. Now you've got to tease that out. Hence, the super-hateful 2 hour coding-and-design session that nobody wants to administer, and nobody wants to experience. Somehow, we've come to the conclusion that at the end of the day, there's nothing that proves someone knows how to work on enterprise-level scalable technology than a whiteboard and a toy problem. What could go wrong?

Finally, assuming *that* works, then you've still got to figure out if the individual is a good organizational fit, or is one of those 'brogrammers' who you really don't want influencing your corporate culture. Sorry - it's a problem. Get past that, and finally you have a potential candidate. If you're counting, you're now down to 1:64 of your candidates (roughly completely made up estimate).

All of us engineers know that after that, we're supposed to hire the 3rd, best candidate (Oxford comma, beware). No problem - we'll just wade through the next 128 candidates...

I don't see a good way out of a lengthy exam of some sort, but I would suggest an annual bar exam, kind of like what lawyers have to do to practice law. This would give engineering HR a known baseline (which we kind of use the BS in Science as right now). The difference between this and a BS would be that an engineering bar can change over time as technology does. It would include a technology-proficiency part, and a data-structures and algorithm component. With that in place, organizations can focus more on corporate culture and recent work experience,instead of lengthy, compressed, coding and design tests.

React-Redux Concepts for API Devs

If you landed here, then chances are you are an API developer who wants to know what this React-Redux buzz is all about.  I'm transitioning biaschecker.org to React-Redux.  I maintain the API and front end technology stacks for this service.

For this post, I'm not going to say a lot about why I chose the stack.  What I'm going to cover are those concepts I, and I'm assuming other, API Devs probably struggle with when looking at the combination of technologies above.

What is React? Redux?
More appropriately, what are react and redux?  This is probably a useful time to get into what they are not.

React is not Redux
Say it again a few times.  Good.  These are separate libraries which do entirely different things.  I'll link the formal docs for both at the end, but here I'll explain a bit.

React is a component definition library.  It's also pretty simple.  You get a React component by subclassing component (basically), and that component gives you some cool features, like only re-rendering when it needs to. I use subclassing loosely, though.  It is JavaScript after all, and it'll be confusing to get into it in a conceptual discussion, so let's say it's close enough.

But...react is not ES6.  Nor is redux. All of those fancy syntactical changes in ES6 are not react, but react really is more digestible in ES6.  Note that if you're a web dev, then you probably laughed just then.  But if you've been knee deep in API or service development like me, you need to know this.

Redux is a way to store state and monitor state changes, if you're a web dev.  A good way that helps me to think about it is as a messaging system, with a subscription model.  As with any messaging system, the format of messages is something producers (your app) and consumers (also your app) need to agree on.

With that system, however, comes a huge list of limitations on how you should use it. These are conventions.  If you step outside of the conventions, people already in the community should, and will, yell at you.  I'll cover these in more detail.

Summarizing, react != redux != es6.  React == components, redux == state and messaging.

One more thing.  These technologies live in Nodejs, and are very disciplined (as seems to be the habit of the node evosystem) with regards to separation of concerns.  This means installing react and redux and even react-redux (bridge lib which makes a lot of things easier) will only ever be enough for test apps.  If you want to do real apps, you'll need a lot more.  I'll link some resources I've used.

Gotchas
Okay, so you've read about actions and reducers, and that there should only be one store. I'll fill you in on what probably wasn't explained.

Reducers only change state, without side effects.  Actions return events which tell the reducers when to change state (and dispatch those events).  So when do I make my API calls?

Well, in the action functions, before returning the event.  The results of those calls return bits of data you add to the event before returning it for dispatch.  It seems simple, but it's actually confusing because the action functions are called from the reducers.  This makes those API calls side effects, yes?  Conveniently ignore that complication.  This is a convention, and actions are the right place.

Why doesn't my state ever change when my react component re-renders?

Too much magic.  You've been swimming in java or c# too long without coming up for air.  There's literally nothing magical about react and redux.  React does what it says, and tries to re-render when it thinks it's time to (glossing over the fact that I'm kind of conflating react and react-dom).  If you want your component to be smarter than that, you need to make it so.

Fortunately, redux exposes a subscribe method (see? messaging) which will notify you when something else in your app changes the state.  Hook this, and look for things in the state that your component knows about.

The biggest hurdle in understanding react, redux, react-redux for me so far, is what I've mentioned here.  Conflation of ES6, react and redux concerns, expecting too much magic, and being a bit too literal in interpreting the associated conventions.  Learn from my mistakes (and ask me anything - I'll respond if I can).

Links:

React:  https://facebook.github.io/react/

Redux: http://redux.js.org/

BiasChecker.org in 30 days for 0$

Facebook and Excessive News Bias

During the months preceding the election of 2016, I noticed a disturbing trend in articles I was seeing posted on Facebook.  Many of my friends (and I'm pretty discriminatory about who my friends are) were re-posting news articles, on multiple sides of political issues, which I thought were clearly biased, or, more specifically, had an objective in the news story, and made very little attempt to hide it.  Memes are the most distinct form of this behavior that I could recall (as an illustrative example), but most articles run the gamut from meme-style blatant bias, to more difficult to distinguish bias articles.  Since I subscribe to the belief that people, once made aware of the bias presented to them, can make better decisions and think more critically, I decided to make it more apparent to people who choose to examine, what the bias is in their news articles.  BiasChecker.org was born from this concept.

Mission Accepted!

The mission was to create a website to evaluate news links for bias terminology and present those bias scores in a easy-to-understand format so that casual users can determine for themselves how much they should trust an article.

Gathering Functional Requirements

In order to present bias in a meaningful way, I had to figure out a way to evaluate for bias quickly, and then present that evaluation in a way that casual users can easily interpret.  Since my target users are on Facebook, I asked the community what they would like, and presented ideas on Facebook to get reactions.  Some of the requirements we distilled out are as follows:

1. Need a way to check links from Facebook.
2. Present the result automatically (if necessary).
3. Share BiasChecker.org results with other Facebook users.
4. Ensure that BiasChecker.org results are in fact useful.
5. Allow individuals a way to check their own news links.
6. Allow individuals to estimate bias on their own (a useful metric to compare algorithmic performance with perceived bias).
7. Auto-monitor individual user's Facebook feed for links posted between App users so that they don't have to auto-import links.

Evaluating for Non-Functional Requirements

Non-functional requirements include the technological requirements and deployment infrastructure.  I needed an environment I could get up and running in quickly, and could still scale quickly (after all, Facebook has tons of users and it's good to be prepared).  I also wanted to work in a technology that was somewhat different than I use every day (to keep it interesting), but that I still had sufficient skill to be productive in.  I chose to use nodejs and Couchdb.  Some of my other technology decisions include:

• Development
• I settled on nodejs because it was different enough to keep me interested, and lightweight enough to do what I need to do without a lot of overhead.  One of the additional benefits was ease of deployment and scalability by using heroku.
• Database
• I settled on Couchdb because much of what I'd be doing is processing, then storing those results, and reporting summaries of those results.  Couchdb views were perfect for this, and I had the added benefit of putting the database in cloudant, IBMs couchdb in the cloud.  This was a really awesome experience, because my last couchdb foray had been 3 years prior, and couchdb has come a long way since then.
• Web Server
• I wanted to be able to present a service very quickly.  With expressjs, I could very quickly set up a web service in four lines of code.  Granted, getting it working past that takes a little more work, but still do-able.
• User Interface
• I wanted to present a very simple UX, using an existing framework, to minimize the UX work (since I'm traditionally a platform dev).  I chose materializecss for this, in part, because I would already be using jQuery, and it makes everything very easy to do.  I've worked with bootstrap.js before, and there was a lot of bouncing back-and-forth between css and js involved there.  I didn't evaluate angular since I've seen angular code before, and there seems to be a bit of a learning curve there I didn't feel like taking on yet.
• Graphing
• I wanted to do things like graphs to display aggregate totals for different tasks.  I've used jsflot in the past, and though I like the robustness of it, I opted for plot.ly instead, because I felt I would only really be doing basic things, and plot.ly has a very simple interface for the types of things (like histograms) I felt I would see.
• Facebook SDK
• I decided to use the Facebook javascript SDK, specifically opengraph v2.8 (which was the default at the time I started).

So far, I didn't need any physical infrastructure except my development laptop.  This was something I already had (as a developer, I've got a few).  Everything I've listed here has a low-cost option. Heroku deploys to dynos, at 0$ for personal use.  Cloudant offers the first 30 days for free.  plot.ly, meterializecss, expressjs are all open-source.

Result

The mission was a huge success!  In less than 30 days of coding and no local infrastructure, I was able to field https://www.biaschecker.org, with all of the features listed above.  I will be iterating on it, but here's an example of what kinds of useful information you can discover about an article using biaschecker:

Landing Page

The only allowed login is with Facebook.  This may change, but given that a lot of the functionality is for Facebook, it may not.

Consumable Bias Score

From a ratio of bias to non-bias terms in an article, and scaled to magnify differences, I produce a bias score between 1 and 10.  1 is the least biased, and 10 is the most biased.  I also offer you the ability to rate an article yourself, and then produce a consensus average of all individuals who have rated the same article.

Bias Term Histogram

The following is an example of what you can learn about your articles.  We show a histogram of all of the biasing terminology we've detected.  The bias terminology comes from a cornell study https://www.cs.cornell.edu/~cristian/Biased_language.html, which conveniently consolidated lexicons from earlier works.

Conclusion

In 30 days, I was able to (with a lot of help and guidance from friends) build biaschecker.org from scratch, and host in the cloud, with no up-front cost.  The recurring monthly costs will be determined by the usage.  Since I don't want to ever charge folks to use this, I also leveraged the new paypal.me feature, to put a link on the site.  Any money donated through the site will be used only for biaschecker expenses.  If you are interested in donating, follow this link to paypal.  Interested in helping out?  Add a comment here and I'll reply, or come message me on linkedin.  The source code is not yet available, but will be opensourced once we have a user base.

Adventures in Databasing

AmandaDB

Some time ago (a year next January), I began working on a database called amandadb.  The goal of this database was to store daily stock values, with the intent on using machine learning techniques to extract meaning out of them.  It wasn't supposed to ever be a "real" database, but there were some practical problems I experienced along the way, which caused me to move in that direction.  For this post, I'm going to share with you some of the creation process, what worked, and what didn't work, to move this project along.  I'm kind of proud at where it is right now, but we'll get there.

Lesson 1: Don't be all things to all people...

Like I mentioned earlier, this was a decision early on.  I had a very simple goal in mind: take (specifically Yahoo) stock tickers daily, and put them in persistent storage.  Then, retrieve those data in a way that's fast  and allows me to do machine learning on them.  At this point, MySQL or SQL Server seemed overkill.  I was literally going to dump the data for each stock ticker into a file on the hard disk, no problem. I got this working within less than a day, and then I realized something.  It took a long time to read those data.  Though I had managed to persist data, I had some serious doubts as to whether it would hold up under the weight of analysis.

Let me clarify.  I wanted to slice and dice different ticker values, over different ranges of time.  I was polling for the entire spread of S&P500 stocks (so, about 500 of them).  It turned out that Yahoo Finance API allowed free download at a rate of multiple times a minute, up to a cap (I think 20k queries daily at inception).  This meant that I could do more than I originally thought (remember, daily only), and I wanted to use all of those data too.  Do the math, and you can see that that adds up pretty quickly.  My plan was to aggregate a year's worth of data, and use that to do some k-means, PCA, genetic algorithm, and eventually perhaps write my first neural network against it.  Finding the specific records I wanted to find was going to be tough.

Just to give you an idea of what I was up against.  Imagine doing essentially a table scan across 20,000 records per day * 365 days per year?  That's 7,300,000 records, being scanned, every time I want to do some data mining.  Ridiculous.

So I hit a roadblock.  Stop?  No way, I was having way too much fun at this point.  Don't be all things to all people was good that I limited my interest to only stock tickers at first.  This allowed me to quickly get to some interesting problems, ones that I thought I already knew how to solve.  

Lesson 2:  When in doubt, add an index...

Or add two.  This is when I started factoring out my code into the first rev of amandadb.  It was because I wanted to add an index.  I realized at that point, that I was basically starting down the path of building my own database.  It still was going to be pretty minimalist - just an index and some files, so no need to get one of the 3rd party solutions yet.  For an index, I just used a sorted dictionary, and serialized it to disk.  This worked well, and drastically improved my access times from something ridiculous to around 7ms per record, when selecting a range of data.  Here's my index in almost it's initial carnation.  So pretty reasonably fast, and did what I needed to do.  I used generics, so I could re-use my index on multiple fields.  I added actually two indexes, one for the date, and one for the stock ticker field.

Lesson 3: Race conditions abound...

I did one other thing at this point.  Whereas before, I just dropped each ticker into its own file, I decided to compile multiple tickers into a single file.  This is so that when searching, I can load up a bunch of records at once.  This introduced a race condition that I discovered while running automated tests. Because the tests parallel where possible in VSTest, I kept getting IO exceptions when writing to the ticker file.  This surprised me, because I wasn't planning at that point on supporting multi-write scenarios, but I had to fix them anyway.

That's when I introduced file locking.  This might be considered the rough equivalent of write locking in the database world.  When writing to the same file, I lock the file, then write, then unlock the file.  I realized that even more, due to the practical evolution of this project, I was writing a database.

Then I discovered value investing.

Lesson 4: Requirements sometimes change...

I read the book The Education of a Value Investor, and that convinced me that applying machine learning to the stock market, though interesting, was basically speculation, and too much speculation makes it difficult for companies with real value, to make that value available for sell.  That meant that I no longer had the initial requirements of limiting the records to stock ticker only.  That also meant that I was in danger of turning my database into vaporware.  I thought for a moment, and pivoted a little bit.  I considered what I had.

A database system that serialized strongly-typed C# POCOs to persistent storage, and a consistent method for retrieving them, using indexes.  If I could get the access times down from 7ms, then I could have a very useful tool on my hands, one that I could actually use in production situations. This tool would fit in that space where we need persistence, but not a full-fledged database server.  If only I could get those access times down....

Lesson 5:  Memory is faster than disk...

So remember those automated tests that I told you about?  To get those to work in a reasonable amount of time, I had essentially mocked out a file system (IAmandaFile, IAmandaDirectory).  I then created in-memory versions of these.  I had already mentally changed requirements to compete with SQL Server access times, so I wanted to be able to store 8KB of data really quickly (< 7ms).  For small records, I was already there.  For large records, not so much.  I was seeing access times of 150ms per record, sometimes.

Combining all of the above, the fix was simple.  Formalize my in-memory access stuff, and use that instead of writing and reading from file all the time.  I was able to completely change this in regards to the record access pretty quickly (since I already basically had the objects necesssary).  This reduced my access time to sub-millisecond, but introduced the requirement to synchronize to persistent storage later.  This seemed familiar to me, so I looked it up.  I had basically migrated to a write-back or write-behind cache.  This is common in many databases, and makes sense if you think about it.  There's no way to get the really, really fast database access times on disk.  This also highlights the benefit of having a persistent flash medium instead of disk, if you can afford it.

Conclusion

So this is pretty much where I'm at.  I'm about halfway through the write-behind cache, and haven't started attacking the issue of synchronization yet, in any real way.  I left out some things I did around record serialization, to avoid seek time in files, and a significant change to my new file-based indexing scheme to allow indexes to take advantage of the in-memory objects as well.  If this project interests you, please feel free to contribute.  At this point, I'm planning to get the write-behind cache fully operational, and then there's a choice to make about next steps.  Being one person, I can either add a service to accept multiple simultaneous writes, or I can add updating capability, literally there is plenty of work to do.  

I think it's neat that I arrived at this point from a very practical beginning, kind of tracing the overall evolution of databases, in a weird way.  There's more to learn here, too - and I'm almost caught up .  I'm planning on using a suffix-tree indexing scheme for strings, which should be really interesting, if it works out.

Grow Your Career with Higher Quality Code

My brother, John, has a degree in Archeology from Texas A&M University.  What this means is that while I was learning about bits, decision trees, and Von Neuman Machines, he was travelling the world to work in Guatemala in the Peace Corps, Africa attached to Americorps, and took the time to become fluent in Spanish, and decipher hieroglyphics on old pyramids.  In other words, he's the cool one, and I'm, well, a geek.  But we have been on many trips together, and (trust me this is relevant) one such trip was to Italy around six years ago.

Distinctly I recall being hurled about in a tiny taxi roughly the size of my highly-fuel efficient Prius, hurtling through the streets toward certain doom (en route to our hotel).  Frantically, John and the driver were having a conversation in a language I didn't understand, his and the driver's hands undulating wildly while pointing at various signs I couldn't read.  I realized they were arguing about which was the best route to travel when the automobile pivoted into a sharp right, barely missing oncoming traffic.

When we came to a stop, thankfully at our hotel and not in the grill of a semi-truck, I tipped the cab driver generously, and asked my brother when he had learned to speak Italian.  To my surprise, he responded that he didn't really know Italian, but that Spanish was 'close enough' to the dialect of Italian the driver spoke that they were able to communicate, nonetheless.  That's not at all the impression I got at the time.  I wonder now, thinking back, why they didn't simply both speak English (as I know the driver had greeted us in English at the airport) as opposed to attempting to communicate in an complex mixture of two different languages and manual gestures.

It has since occurred to me that coding is a lot like verbal communication.  The language used is similar to a cultural vernacular, and comments are analogous to those hand-wavy exclamations used to get certain points across which aren't clear from the language itself.  Sometimes, it's enough to get a feel for the code and interpret a few gestures, but usually, actually speaking the same language (or dialect) is preferable.  Also, there are some things you just can't say in one language or dialect versus another.  In code, this might be analogous to why someone decided to use nodejs for building a service, instead of WCF (or vice versa).

I'll come back to analogies to verbal communication throughout this talk to try to influence your perspective, as I attempt to demonstrate that code quality matters.  However, I will always defer to the number one rule: get code to work.  But once code works, what next?  I will explain why I (and other managers I know) pay attention to things like code quality when reviewing code, and what different transgressions in style or pattern mean to me, as an Manager of engineers.

My talk on this subject is roughly an hour long.  Here are the high points of what I'm going to be discussing.

1. Intro
2. Code is Communication (with examples)
3. Code is Design (with examples)
4. Code is your Career
5. Questions

After the talk, I'll be posting my slide deck here with examples and explanations for your reference :).  My goal is to help improve your coding career by giving you a perspective on what your coding choices, style, design say about your development as an engineer.

This camp will be on Saturday, 9/10 from 3:30 to 4:30.  I'll be there most of the day, so if you see me drop by and say hello!

Chuck Sweet's Top 5 Rules for Software Engineering

Recently, I've been thinking about how to keep myself out of trouble with regards to software engineering.  I'm a huge fan of greedy algorithms (simple, straightforward), and kind of related, the KISS principle.  Don't get me wrong, I can write some crazy complex software, but I'm past the point in my career development where I feel like the complexity of code I write has any relationship to my standing as a software engineer, except perhaps an inverse relationship might exist.

Anyway, while thinking about how I approach software engineering, I've come across these rules which I apply every time I write code.

Rule 0: Performance is king.
If I don't write performant software in the product that I'm supporting, someone in a competing product will. Features matter, but if folks can't actually use my product because it performs too slowly, then eventually, your company will be lost to those whose tools they can use.  Say it with me:  "Performance Is King".

Rule 1: Existing patterns should not be changed for light or transient causes.  
In other words, if I'm new to the project, and I don't like the patterns, don't change them.  In fact, my new code should look remarkably similar to the existing code I find.  This is to combat a tendency I have to rewrite more than I need to.

Rule 2: Design twice, code once.  
This is the rule which keeps good design firmly in place, regardless of development methodology.  It doesn't matter if we choose agile, extreme coding, six sigma, whatever new fancy methodology doesn't exist.  If our DNA makes us design twice, code once, then we can avoid the problems I describe here.

Rule 3: Don't trust your own code, ever.  
I could have named this rule 'test, test, test'.  It never fails that I'll get my code to a state where I feel like I'm ready to ship, and I've dotted my 'I's, and crossed my 'T's.  Then I'll write a test (if I didn't start with a test, as I sometimes do), and the code immediately explodes.  I was writing a multi-threading client that I got to this state, and writing and running tests actually proved that I couldn't kill the threads once I'd spawned them (coded the way I had).  It was simple to fix, but it wasn't lost on me that, in production, I would have toppled servers, with no recovery except to restart them, and potentially huge customer implications.

Rule 4: Code simpler.  
This requires a little bit of explanation.  If you're like me, and wrote a bunch of code, then there's a possibility that it's over-engineered.  My next step these days is to consciously review my code, and see what I can remove, or code that I can't really justify.  This dovetails nicely with the extreme programming YAGNI principle.  This is a reminder to me that when I think it's simple enough, make it simpler still.  As a quick aside, patterns are good.  I've mentioned this before.  The problem with patterns is that they're often pretty robust, and offer a lot of extra stuff  just in case.  That's another place this helps.  Maybe that extra interface isn't necessary- would a regular class work?

Rule 5: Trust other engineers.  
This keeps me out of the NIH trap (not invented here).  It also means that if I'm writing a class, and have a property (or method), I default to public accessibility, unless there's a solid and compelling reason to be less accessible.  This one  rule reminds me that even though I've been coding for 20+ years,  it's possible that I don't know everything.  This is why I attend meetups like the Seattle Scalability Meetup, and other events to chat with other software engineers, and keep an eye on the evolution of software engineering.

These rules are designed to catch personality hangups I have seen in myself, and so may seem mundane to you if you don't have the same personality snafus.  I write them down here, so that just in case, maybe, someone else has the same tendencies, and will get some benefit.  Happy coding!

This Is Not The [Evolutionary] Design You're Looking For

Let me caveat this by saying that I'm a fan of some of the more recent development trends like TDD,  Agile, and what these changes have done for software engineering generally.  As a software engineer, I know first-hand the temptation to code for every eventuality, especially when they seem so obvious sometimes.  These types of approaches help make sure that the developer stops developing at some point by providing a terminating point.  In TDD, you start with a degenerate unit test, code until the test passes.  Then you incrementally add to the test until you match what's in the requirement (and no more).  Once that final test passes, you stop writing code.  In Agile, the team agrees on what something called the MVP (minimum viable product) is, and commits to delivery in a sprint or two.  That time-boxes everything so that the there literally is not time to do much in the way of over-engineering.

In spite of the massive cost-saving benefit of making sure that we software engineers don't stray off of the beaten path, there's a down-side.  That down side is this: when are you supposed to do system design?  I would suggest that with competing priorities, evolutionary designs simply don't work in an organization.  This shouldn't be a surprise, as Martin Fowler suggests that evolutionary design, as done in practice, is actually done in a way that leads to 'chaos' [Fowler, Is Design Dead].  I've seen this play out in such a way that any attempts at refactoring were actively discouraged by the senior engineers and developers in an organization!  When that happens, any chance of a reasonable design evolving is definitely dead.

Assuming that's a worst-case situation, what actually happens in real-world software engineering?  A product owner or some customer advocate presents a set of requirements, which then the software engineering team is responsible for delivering.

In the best case, the team can do an estimate, and then propose the estimate to the product owner.  Let's be realistic and assume that the estimate is actually an under-estimate by, say, 50% (that is to say, the actual work will take 150% of the LOE suggested by the engineering team).  Simply put, the engineering team says 2 sprints, the actual LOE is 3 sprints.  At the end of those 2 sprints, there's another feature that the team is expected to start work on.  Then, when does this alleged design work happen?  At the end of 2 sprints, the team is realistically still finishing up with the previous work, and so has no actual time to do design work.

In the worst case, the product owner has already committed a delivery timeline to the customer.  This example doesn't require a lot to show that design won't actually happen.  The engineering team is not in a position to do any real design, and has likely started out behind schedule already.

The lesson here is straightforward:  the only time we engineers have to do real design is time that we carve out for ourselves.

To conclude, there are many forces acting against thoughtful design in the newer development methodologies.  I've specifically examined Agile as it is often implemented, but you can read the same thing in Martin Fowler's discussion on XP.  What he closes with, and I wish to re-iterate, is that without the will, design will simply not happen - there are too many forces pushing in the opposite direction (probably as a reaction to the design-heavy requirements analysis - not going to bother to complete this argument though).  So, as engineers, we can't assume that good design will arise by KISS, YAGNI, DRY, or any other cute phrases to help us simplify.  We must intentionally bake design into our DNA.