379: Feature Flags

The Bike Shed

• Episode: 379: Feature Flags
• Website: The Bike Shed
• Feed URL: https://www.bikeshed.fm/rss
• Duration: 00:41:56
• Published: 2023-04-11 06:15

Joël submitted a last-minute submission to RailsConf discreet math, which got picked up! 🎉 He'll be speaking at RailsConf 2023 in Atlanta at the end of April about why it's relevant to developers and all the different practical ways he uses it daily. Stephanie recommends headlamps for in-bed reading sessions and sets up the feature flags topic for today based on a project that must be released to the public in one go. This episode is brought to you by Airbrake (https://airbrake.io/?utm_campaign=Q3_2022%3A%20Bike%20Shed%20Podcast%20Ad&utm_source=Bike%20Shed&utm_medium=website). Visit Frictionless error monitoring and performance insight for your app stack. Joël's Railsconf talk (https://railsconf2023.sessionize.com/session/453592) Episode on Discrete Math (https://www.bikeshed.fm/374) Episode on the Fundamentals (https://www.bikeshed.fm/371) Strangler fig pattern (https://shopify.engineering/refactoring-legacy-code-strangler-fig-pattern) Flipper gem (https://github.com/jnunemaker/flipper) Transcript: STEPHANIE: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn. JOËL: And I'm Joël Quenneville. And together, we're here to share a bit of what we've learned along the way. STEPHANIE: So, Joël, what's new in your world? JOËL: So a few episodes ago, we had a guest, Sara Jackson, on and she was talking about some cool elements of discrete math and how those can be really practically useful to us as developers in our day-to-day work. I was really inspired by that conversation. And the day we recorded that, I think, was the last day that the RailsConf CFP was open. So I went and submitted a last-minute submission to RailsConf for this idea, and it got picked up. So I'm going to be speaking at RailsConf 2023 in Atlanta at the end of April about discrete math and why it's relevant to us as developers, and all the different practical ways that I use it on a daily basis. STEPHANIE: That's awesome. Congrats, Joël. I'm so excited for this talk. JOËL: Thanks. STEPHANIE: Was this an 11:59 p.m. submission? JOËL: Aaah, very close to that, yes. I don't recommend people to do this. But if inspiration hits you on the last day of the CFP, do it; go for it. I'd actually submitted two talk proposals. And I think maybe a little bit of the excitement in the energy for this last-minute one came through because that's the one that the committee picked. STEPHANIE: How did you know that you would want to turn this topic into a talk? JOËL: I think because I was excited to share about this to other people, and we'd kind of already done that on the show. But this is the sort of thing that's like, you know what? I'm kind of feeling a little bit of fire of this idea. I think more people should know about this. I think there's value in sharing this idea more broadly. What are some areas that I could do? I could maybe write a blog post. Oh, RailsConf, that's open until tonight. Let me put together a proposal because I'm excited about sharing this idea. STEPHANIE: Yeah, I think we chatted a little bit about discrete math as potentially a fundamental pillar of information or a skill set for developers back in our episode on just the fundamentals of what developers need to know. And I was just thinking as you were talking about it that that was an audio medium, but it's obviously kind of an academic topic. And so to have slides for it and almost make it kind of like a little mini-lecture but more fun [laughs] than sitting in a classroom, I think, could be a really cool application of this topic. JOËL: Yeah, I'm excited to take a very practical look at this. I've got 30 minutes to talk about it. I'm not going to give you the deep mathematical fundamentals. But if there are little bits of discrete math that I can show that are actually practically useful in a day-to-day situation, I think that would be incredibly valuable. So currently working on the talk, but the way I have it structured right now is very scenario-based. So we're going to look at some problems, look at just a tiny little bit of discrete math theory, and then see how that allows us to solve the underlying problem. STEPHANIE: Nice. I won't be attending RailsConf this year, but I really look forward to watching the video when it comes out. JOËL: Thank you. So, Stephanie, what's new in your world? STEPHANIE: So last night I was reading in bed, and I usually stay up a little bit later than my partner. And so he was going to bed, and at that point, I usually turn off the bedside table lamp and either move to the couch to keep reading or just go to bed as well. But I was really engrossed in my book, and I wanted to keep going. But I was so cozy in bed. I really didn't want to move to the couch because then you risk falling asleep on the couch, and nobody likes that. And so I was like, oh man, you know what I would really love? One of those book lights. And I got this really vivid memory of the clip-on ones I used to have as a kid that would clip onto the pages of my book and shine a light onto the page. And I was like, I feel like that technology must have gotten better [laughs] since I was a child. And I was like, okay, now I need to look up some doodad and buy a thing to help me solve this problem of reading in bed after hours. And I was scrolling through my phone for a little bit, but then I was like, well, I want to read right now, not after [chuckles] I've purchased this gadget. And it hit me that we have headlamps that we use for our camping trips and also just if we ever are stuck without electricity. And I grabbed one of those from the junk drawer, and I wore it to bed and was reading my book. And it was a pretty good experience. So I'm actually pretty happy that I didn't need to buy a new thing to solve this problem of mine. And yeah, it turns out that you can just use something that you have at home. JOËL: That's really cool. I feel like when I've used the headlamps before; normally, they're incredibly bright. Did you find that to be a problem for you? STEPHANIE: I was able to adjust the brightness on mine, so it was definitely on the lower end of the setting and, overall, just better than having the nightstand lamp on, which I think just totally brightens the room. And at least this was a more targeted area of light, and it's working so far. He didn't wake up or anything, so I would call that a success. And now I have it in my bedside drawer. And I think I look really silly, but that's okay. No one can see me anyway. [laughs] JOËL: That's kind of the whole point of it, right? STEPHANIE: Exactly. [laughs] So, aside from my bedtime routine, another thing that's new in my world is something project-related. So I wanted to bring this up to you and get your thoughts on the situation. So I want to talk about feature flags because I'm currently working on a pretty big project that has to be released to the public in one go. And so naturally, we reached for using feature flags to be able to release our work to production but not to make it accessible to users so that we could be working on this thing incrementally and not have a huge release where all of this code goes out. But as we've been building the feature, I am realizing that we are having to plug conditional code to check for this feature flag in a lot of different places. So, so far, we've been putting it in controllers. We've been putting it in a menu builder to show the user where they can navigate to. And then we've even been putting it inside of methods to change behavior based on whether the flag is on. And so that was kind of, I think, getting my spider senses a little bit tingling. And then recently, one of the bigger issues that our team had a discussion about was whether to include this conditional check for the flag in queries. So we are building on top of an existing model. But once the feature flag is on and customers are using it, the application will be able to create new records for that model. But they're of a different category with a different value for one of the attributes. And we end up clearing for this model in a lot of one-off places. And so once this thing is on, all of the records that are specific to this new feature will be included whenever we query for this model. And as we've been developing, it's been less of an issue because customers can't access the flow to create these new records. But someone brought up what if we release this, and it turns out that we have an issue and we want to turn it off; we want to roll it back? And at that point, those records will still have been created in the database and will then be included in those queries and what do we do then? And so we started getting into the weeds a little bit of, like, do we want to have some conditional query situation going on? We haven't quite landed on an answer, but things are getting a little hairy. And I am curious if this works. Any thoughts for you? JOËL: That's a really interesting problem to have. You mentioned having all of these conditionals kind of sprinkled everywhere for the feature flag kind of triggered your spidey senses a little bit. Do you have a sense a little bit of maybe why that feels wrong or maybe why you're feeling uncomfortable about this code? STEPHANIE: Yeah, I was thinking about it, and, in my opinion, the ideal world for feature flags would be you have the checks at the boundaries of your code or where a customer could interact with the application. And then you are able to, I guess, branch a little earlier so that they don't go down the flow at all where the changes are being made. And this seems to be a little bit of the opposite, where we end up having to check in a million different places because we aren't keeping that separation as explicit as I think it should be. JOËL: Hmm, so almost like it's not the feature flag that's the problem. The feature flag is just a symptom of tight coupling in the wider system. STEPHANIE: Yeah, that's definitely a smell that has emerged. But I also don't think that we have the luxury necessarily to decouple all of the places in the code as we are trying to add this new feature. Have you been in a situation like this before? JOËL: I think tight coupling, in general, is a thing I've seen in a lot of projects. I can't immediately think of a moment where it was highlighted by introducing a feature flag. Sometimes I think what can be tricky is if you have a feature that has a lot of cross-cutting concerns, then it's easy for that to kind of bleed into other things. There are a variety of techniques to try to, like you said, isolate the new code such that you don't need to conditionally branch on it everywhere. I wonder if there might be maybe a few important inflection points that you could introduce some sort of wrapper or push a conditional higher up the decision tree or something like that that might get 50%-80% of the way there and at least eliminate a lot of the pain. STEPHANIE: Yeah. I like the use of inflection points because I think that right now, our strategy has been everywhere as we are making changes, just putting it in as a guard when there's probably some degree of higher level thinking about, okay, we're now changing a bunch of internals of this class, but maybe the change that we're making is its own concept. And it could be a separate class, and that is when we choose to use that class like that as an inflection point. JOËL: I have this vague mental model in my mind of building a mechanical system and wanting to know where do I want it to be rigid and where do I want some sort of joints so that it can flex? And I don't want the entire system to be made out of joints because now it's a pile of spaghetti. It's important to be rigid in a lot of places. But in some places, I do need it to flex. And it's identifying what are those right places where it needs to bend to flex to different scenarios? And to me, that's a metaphor for when do we need abstractions, the ability to choose between different paths, the ability to maybe have some polymorphism? And when do we want to force everything to say, "No, all the code is going to behave in this one way?" STEPHANIE: Yeah, that's a great point. I am also kind of stuck on what you said earlier, where the feature is very cross-cutting, and I think that is true for this project at hand. And so that has been very challenging too. There's also a lack of trust that it will just work with the other parts that it's touching because we're actually extending something that was already a bit of a unique flow or a kind of a one-off situation. And it was built very particularly to support that one thing. And now here we are introducing something else for it to support and having to go in and change all of those individual places where we were making those one-off exceptions to now also be able to handle this new thing we're introducing. JOËL: I think, in many cases, I appreciate when people write code in that way and that they didn't try to abstract everything on that first difference. But now that I'm the one coming in doing the second one now, it's time to think, okay, clearly, we're trying to flex here. We probably need some sort of abstraction. Unfortunately, that means it does have to come into the time budget for this feature and say, look, we already have a special case here. Now we're going to add a second special case. We need to take the time to do a little bit of refactoring or a little bit of abstracting in order to make that work properly. STEPHANIE: Yeah, absolutely. I think the logic there originally was that, oh, we already have a special case for this thing. So it should be easy [laughs] to now add this other special case that's kind of based off of the original exception. And I actually think that worked against us a little bit because I'm mentioning we're introducing this new feature flag, and I'm also seeing remnants of that old feature flag that didn't get quite cleaned up as well, so just a lot more complexity in the different ways that the flow is going. JOËL: In some ways, you kind of see earlier failures with this exact same approach that you're trying to take. So it's a bit of a warning. STEPHANIE: Yeah. I want to avoid leaving those traces for future developers. There's definitely some degree of regret that I feel for all the times that I've introduced a feature flag and never cleaned [laughs] it up. So seeing just remnants of this same approach, like you mentioned, and wanting to do something differently to make sure that we aren't creating as much trouble for the next time around. JOËL: So you'd brought up cross-cutting concerns earlier. And I think some common situations for that is like, oh, we've got some code that changed in the model where we have to put conditions. But then we also have a parallel condition in the controller and a parallel condition in the view, and maybe even in the routing file, and it quickly becomes a mess. Sometimes you're able to sort of take all of those, and kind of put them in a different part of the app. So you might have a view file that is really complex and has all these conditionals. And maybe you kind of give up trying to make it all work in one file and say, you know what? For this special case, we're going to do it all in a separate view file. And maybe there's one top-level condition that says, if this thing is true, render the second view file; otherwise, render the first one. And now you've only got one condition instead of having it kind of sprinkled all over some other piece. STEPHANIE: Yeah, that actually is the approach we ended up taking for this project. And in this case, we have React on the front end. So we're rendering a completely different React component based on the flag. And that did solve a lot of the front-end-specific complexity. And still, I think we are seeing a lot of the coupling be an issue on the back end, which is interesting. MID-ROLL AD: Debugging errors can be a developer’s worst nightmare...but it doesn’t have to be. Airbrake is an award-winning error monitoring, performance, and deployment tracking tool created by developers for developers that can actually help cut your debugging time in half. So why do developers love Airbrake? It has all of the information that web developers need to monitor their application - including error management, performance insights, and deploy tracking! Airbrake’s debugging tool catches all of your project errors, intelligently groups them, and points you to the issue in the code so you can quickly fix the bug before customers are impacted. In addition to stellar error monitoring, Airbrake’s lightweight APM helps developers to track the performance and availability of their application through metrics like HTTP requests, response times, error occurrences, and user satisfaction. Finally, Airbrake Deploy Tracking helps developers track trends, fix bad deploys, and improve code quality. Since 2008, Airbrake has been a staple in the Ruby community and has grown to cover all major programming languages. Airbrake seamlessly integrates with your favorite apps to include modern features like single sign-on and SDK-based installation. From testing to production, Airbrake notifiers have your back. Your time is valuable, so why waste it combing through logs, waiting for user reports, or retrofitting other tools to monitor your application? You literally have nothing to lose. Head on over to airbrake.io/try/bikeshed to create your FREE developer account today! JOËL: So you'd mentioned queries and how...it sounds like there's maybe an enum in the database that you're adding a new state to, but you want to exclude that new state from any existing queries. I almost wonder if this is a situation where you want to create a sort of no-op commit that could go out without the feature flag, and that would basically prevent all the existing queries from using this new enum. And maybe it introduces the new enum but behind something that blocks them from being used in the existing queries. Depending on how your queries are structured, maybe it's just in your main query; you have a where not extra line that excludes the values in that enum. If it's not nicely scoped, maybe you need some sort of default scope on the model to say, exclude anything that has this enum value. And now you can introduce that enum, and you can have all those records. And even if it's not behind a feature flag, everything will continue to work exactly as it does today, and that's your goal. And that is a fairly small scoped change. It's almost like a refactor in the classic sense of it, the make the change easy before you make the easy change. Although, in your case, it's maybe not that easy of a change. But then, after that, you can build on top of this with your work that might be behind a feature flag. STEPHANIE: Yeah, I like that approach a lot because that ensures that the existing behavior will continue to work as expected. And then that also is a good way to audit, I guess, all the places that we will need to consider when we are making the change. So we already have in mind all the places that things are touching, and it's not a surprise when we find out, oh, we missed this query, or whatever. I was also thinking about...I mentioned that we are querying one-off quite a bit with different filters. I was also thinking that maybe a query object could be a good use case here and wrapping existing business logic in a meaningful query. And then perhaps the new enum that we're introducing would have its own conceptual meaning. JOËL: When you talk about one-off queries, are these like custom queries built out directly in the view or in the controller? Or where are you seeing these one-off queries? STEPHANIE: Unfortunately, it's both the controller and in-service classes. [chuckles] JOËL: Okay. I was afraid you were going to say the view, which brings me back to my old PHP days because I definitely wrote queries in...I guess my entire app was basically the view at that time. I had just one PHP file, and HTML, SQL all went in there. The page submitted to itself. So there's a giant conditional that split the files like if POST versus if GET. STEPHANIE: Oof, that sounds pretty gnarly. And I feel like; in that case, you don't have the tools for the flexibility that you would have liked. JOËL: It turns out the frameworks are nice. It's a good thing to have. STEPHANIE: Good take. JOËL: I think going back to something we were talking earlier, we were talking about how to maybe incrementally ship parts of this, and the sort of no-op approach that I was talking about, I'm a huge fan of. I think it's similar to what's sometimes called the strangler fig pattern that allows you to sort of incrementally sometimes change over from one system to another. But I think a lot of the ideas from that pattern can apply to adding new behavior where you might want to start by introducing some sort of no-op system and then maybe conditionally branching within that and keeping things separate to then evolving to your final piece. And there was an article that I am a huge fan of by Adrianna Chang on the Shopify Engineering blog about this pattern. We'll link it in the show notes. But for those who are interested in digging into this pattern more, kind of wondering how it applies, I recommend reading this article. STEPHANIE: I'm really glad you mentioned that because the other thing I was thinking about that feels like such a trap with feature flags is that it kind of conflates two different things, at least in the way that I've seen it used on different teams where we are using it to hide work that we are incrementally shipping. And then it's also used as a deployment strategy. So maybe you might turn it on for X percent of customers, or you might just turn it on to 100% for everyone. But if something goes wrong, you can quickly turn it off. And for some reason, I'm thinking that those two things...maybe I'm not sure that one tool is the best thing for both of those concerns. JOËL: Yes. Sometimes feature flags are used to actually gate a feature. I tend to bucket the turning it on and turning it off as the same thing as turning it on for 50% of people. But sometimes, it's really used more for a kind of CI/CD process where you want everything that gets merged from a PR to immediately go out to the production server. But you might not be ready to have that particular feature get turned on just yet. But a lot of those things might get turned on pretty quickly, and then the feature flag might get removed fairly quickly as well. So they're not really features in the sense of product people are going to want to be like, okay, now we're ready to turn on this new feature for the big release day. They're very logistical. And they're really almost like another way to do version control but at the release level rather than via Git. Does that sound like a reasonable distinction to you, or is that different than what you're talking about? STEPHANIE: That's really interesting. I think you may have found a third [laughs] way that people use them because I have definitely seen that as well. I think for me, what you said about if you turn it on to 50% of people, that is essentially turning it on to 100% in the sense that if you are trying to mitigate risk and things go wrong, things are still wrong, and you have to figure out what to do then. And I'm glad you mentioned the strangler fig pattern because I think people on teams that I've been on at least have been using feature flags as a crutch for managing risk when they deploy and thinking that, oh, if something goes wrong, then at least we have this mechanism to stop the bleeding. Whereas if you were making sure that the code you are writing is in production but doesn't change the existing behavior because of the way you wrote it, that is a lot more resilient, I think, than just opening the gates and hoping for the best and having a safety net of turning it off. JOËL: I see what you're saying now. And I think you've hit on a really important thing. So when you're trying to build software incrementally, there's a bit of an anti-pattern where you go off on a long-running branch, and you build, and you build, and you build. And hopefully, you're keeping it in sync with the main branch, but maybe you're not. And at some point, there's a big merge, and it's scary and potentially dangerous. And then there's a release, and hopefully, everything goes well. And we generally agree that that style of development generally does not lead to the best outcomes. So we try to work incrementally. We try to merge back into the main branch as much as possible. But I think that many developers are kind of wired to think in that long-running branch approach. And so they kind of try to recreate it with feature flags. So it's like, oh yeah, technically, we're merging, like, we're on small branches, and we're merging back into main, but it's all gated behind this one feature flag, and then it's going to be a big bang release. And it may or may not break, and it's going to be scary. And it's going to be a lot of risk that we're trying to mitigate. But the more we're piling on behind this feature flag, the higher the risk becomes. So it ends up being kind of a long-running branch. STEPHANIE: Yeah, that's a really great connection. I think that is where some of the fear is coming from too. And we are thinking that, oh, we're relatively safe because we have this mechanism that at least we can react with very quickly rather than having to revert a change or something like that. But the way that we're using it on this project...and in general, I've seen other people do it this way. And that could be why this technique has proliferated a little bit. It is a bit concerning; I think, because it still is just okay; we're just going to release it and then hope for the best. JOËL: I think there are valid use cases for that kind of strategy, but I would like to see those come from a product perspective rather than a development perspective. So maybe as a business strategy, we decide that this series of features, we want them all to kind of show up at the same time. They're part of a larger package of changes that we want to release all at once. Now you're getting into product strategy. Oftentimes, it's better to release small things incrementally. But there are some times where you want to release a big thing that has multiple kind of sub-features all at once, and I think that's totally valid. Where I'm less comfortable with it is when it's coming from the developer side because it's more about, oh, we just want to protect all of these things and kind of release a massive change all at once. And that, I think, ideally is better managed by truly working incrementally with something like a strangler fig pattern. I think that manages to mitigate the risks while also avoiding some of the messiness that comes with the feature flag here. STEPHANIE: Yeah, I think the other way of working that I've noticed with feature flags gating larger features is having a QA team member manually going through all of the features behind the feature flag to make sure things are working as expected. But that is still different from real users using it. I think that is another form of reassurance that folks think we're getting with this strategy. But I think it's still limited in covering a production use case which is ultimately giving you the truest [laughs] reassurance that your application is working as expected for the people that are using it. JOËL: I think I'm fine with having a QA person turn a feature flag on on a staging server, try something out as part of the workflow as long as these features are generally kept small. And then we ship it, and we turn the feature flag on, and then maybe we remove it as part of the process later on. Where I get less comfortable is where you've got a team of people working for weeks behind a feature flag. And, again, I do understand that for product reasons, there are times where that's a valid thing, where there's a big set of changes that you don't want your customers to see. But if it's for developer reasons because we're concerned about coupling or we're concerned that something could break, then I get really uncomfortable with saying, okay, we're going to have a team of people working for several weeks behind a feature flag, and then we're going to need multiple QA people to test a ton of work that's all behind a feature flag. Now you're just kind of creating new risk while also trying to mitigate it and it kind of turns into a sort of weird zero-sum game. STEPHANIE: Yeah, I know what you mean. I think in those cases, too, you're only testing what you know. So you're only testing the edge cases and the different flows that you know about. And inevitably, customers are probably using the app [chuckles] in ways that are completely unexpected, and no amount of testing all the different cases that you think are comprehensive will account for the way that it's used by customers. JOËL: To be fair, I think a good QA person will catch a lot of weird edge cases. They know a lot of the ways that customers will try weird things. They often are really good at not being boxed into product or development's ideas of how the product should be used rather than how it can be used. That being said, real customers are a surprising and creative bunch, and they will absolutely do the unexpected. STEPHANIE: Yeah, that's a great point. I'm glad you mentioned that. JOËL: We've talked about a few different ways where maybe you're uncertain or uncomfortable about feature flags. Are there ways of using feature flags that you are 100% in favor of or that you're excited about, like, yes, feature flags are a great use in this scenario? STEPHANIE: First of all, I think one of the most delightful times that I've had with a feature flag is when we use the Flipper gem. At a previous company, I remember really enjoying that one over...I've seen more hand-rolled implementations of feature flags, but I do want to give that gem a little shout-out. I think they are most effective when you do have a new page or something that has a very clear boundary, and you want to make sure that no one can go to the URL and see the new features that way, at least that's like the most clear cut use case for them. And then, ideally, you have implemented everything. You have tested behind the feature flag. And then, the day of the release, we can just turn that thing on, and all is well. I hate to be that person to say simpler is better, but sometimes I think that is true. What about you? Do you have any instances where you felt a feature flag was really effective? JOËL: I appreciate feature flags in more traditional CD, continuous deployment-style systems, particularly once the team grows a little bit and you've got a lot of PRs going out. And you want to be able to maybe undo things without necessarily having to revert and redeploy code, or if you've got things where you don't want to have Git be the thing that's gating when customers get to see features. So maybe a person implements it on a particular day, but you don't want customers to see it until next week because that's when your marketing has said, if we're going to turn it on, do you really want that to sit in a PR for a week and then hit the merge button and that's going to be what ships it out to people? Like, that gets kind of clunky. I think you can get away with it when you're a small, fast-moving team. It doesn't work once you start scaling a little bit. And so I find that at that point, starting to introduce feature flags has been really helpful on teams I've worked on. And I've been part of that transition where we're feeling a lot of pain, and then we introduce feature flags and can move to more of a continuous deployment where you get your code reviewed; you merge it; it's good. And then it goes out, and at some point, we turn the feature on, and then maybe at some point, we come back and clean up the feature flag and say this is now permanently in. I've seen good success with that style of approach. STEPHANIE: Yeah, you bring up a good point about incorporating it into the CI/CD process. And that also makes me think, ideally, we would have some information as developers about how long we want this thing to be in that feature flag state because if you know, then you can claim ownership over it and make sure that it gets cleaned up. I think right now, what I've seen is we are using it in the development process, but then we're not necessarily communicating with product about how long it's supposed to stay this way and what are we supposed to do with it afterwards. A lot of that gets lost. JOËL: It is interesting that I think some feature flags are mainly product-focused, and then others are entirely developer-focused. In fact, maybe the product team doesn't even need to know that this is internally gated by a feature flag because it's more about internally how you manage the release process and continuous deployment. The fact that maybe physically pushing the code onto the server and having a feature be available to users is not exactly the same action might be a distinction that doesn't really matter to your product team. And if that's not a thing that they care about toggling for users, then that can be something that just stays internal to the dev team. And you say, okay, we're using that to manage risk or to allow us to merge code earlier, whatever the thing is, but then we clean them up ourselves, and nobody needs to know. Whereas some things are very much product-focused, and we say, okay, the product team wants this to be a thing they can turn on and off, or they want to run an experiment, and then they very much need to be in the know. And I think you were alluding to some feature flags that might be kind of halfway in between that. They're not clear-cut product ones, but they're not entirely just mechanical deployment flags. They kind of sit in that weird zone between the two. STEPHANIE: Yeah, absolutely. I think maybe there is a case to be made about this distinction, and people can then better use this tool and apply it to the actual problem that they're having. JOËL: I think it's helped me get a better feel for them by having a bit of that distinction in my mind, thinking of flags as these are either sort of mechanical developer flags, or these are products flags. And maybe there's a richer nomenclature that can be developed. But I think at least having that distinction has helped me think about these in a more structured fashion. STEPHANIE: Yeah, and even communicating that in the name of the flag, too, I think could be really valuable for other developers who come across it in the codebase. As for cleaning them up, I was just thinking about how it can be such a pain to then unwind the logic of the feature flag [chuckles] when you're removing them to figure out, okay, now that this thing is done, how do we actually want it to work? I think sometimes that ends up being a bit of a pain point and is what leaves them hanging around. JOËL: And then it's not obvious which of the two branches in a conditional should be kept. STEPHANIE: Right. And maybe that also depends on that distinction where you're talking about between a product release flag and a more internal development tool. JOËL: Yeah, I could definitely see a more product-focused one where it's possible that you might want to keep both and say, no, this is a thing that we want to be able to toggle on. In general, for the foreseeable future, don't remove this flag. I want to control it from a dashboard. Maybe we want this to become maybe not gated by a feature flag in the traditional sense, but we want to tie it to a subscription level on a user's profile. So keep all the conditionals in place, but now just branch off the user's subscription status rather than a global flag. STEPHANIE: Yeah, it turns out that feature flags can be really complicated. It's not just kind of a binary is it on or off [laughs] situation. It almost makes me wonder if some metadata would be helpful with flags to signal more information about them in the codebase. Like, are they a part of your application domain or not? It kind of gets a little fuzzy. JOËL: Definitely. Something could be experimental; something could be like we said, mechanical; something could be deeply integrated into the vision of the product where you know that you're going to want to branch in the future here for different types of customers. I've even been on projects that were multi-tenant, where we had just standard feature flagging setup, but different tenants would have different sets of flags turned on. So they were global configuration for their instance of the app, but each instance had their own unique config of flags turned on and off. STEPHANIE: That's really interesting because, in some ways, flags are a bit of a configuration. But then, when you are combining a lot of different flags to configure something, then maybe that is better done with a different approach. JOËL: I like doing that kind of thing with flags rather than configuration because then it can be done via an admin dashboard rather than having to make changes to a YAML or JSON and having to commit it. And in this particular projects case, it was really nice because it meant that we could sort of, again, find those flex points and say, okay, we know that some clients want things to behave in two or three different ways in this one particular place. We'll gate that behind a flag, and then you can enable the variant that you want for each client. And now the customer service reps can manage that, and not the dev team. STEPHANIE: Got it. Yeah, that is pretty cool. JOËL: So, in a way, feature flags can be a great way to empower other departments within the business instead of kind of centralizing control within the dev team. STEPHANIE: Yeah. I suppose you just provided another use case for them. [laughs] JOËL: I feel like there are probably some really interesting elements to dig into around the interaction between product design and the actual development of things and teams that are doing either business development or customer support. Or maybe just some kind of rep for a big client is sort of where a lot of the power to actually execute things is. That is not this episode today, but I think that could be a great follow-up episode to do sometime. STEPHANIE: Yeah, that's very interesting. I would love to talk about that in the future. JOËL: The short version is I'm a fan of empowering other people. STEPHANIE: There you have it, folks. [laughs] You heard it here first. Joël is a fan of empowering other people. On that note, shall we wrap up? JOËL: Let's wrap up. STEPHANIE: Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at [email protected] via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeee!!!!!! ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.