Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Ep. 1: The line between migration and…?
444 Plays1 year ago
In the first episode of the podcast we explore the boundary between database migrations and other operations that need to happen when the database is being migrated. How do you seed or manipulate data after new tables or columns have been added? In migrations? In one-off commands that you have to run manually? Running seeders in production? In tinker, Nova, or TablesPlus? We spend a whole hour talking about a topic that most people decide on in a few minutes.
Items discussed on show:
Recommended
Transcript
Handling Trivial Developer Tasks
00:00:06
Speaker
All right. Welcome to over-engineered, the podcast where we asked the very important question, what's the absolute best way to do stuff that doesn't really matter? So a while ago I had an idea to do this show. Basically there are things that we run across as developers that
00:00:27
Speaker
every time it's like, well, it doesn't matter. Just kind of, we'll just do the thing that gets it done this time. And every time it feels like, well, it doesn't matter. We'll just do the thing that gets us there this time, but it keeps on bugging me. Like someday I want to come back to this. And so I decided what the heck, we're going to get people together. We're going to argue about these, these little tiny details until either
00:00:53
Speaker
we're really sick of arguing or we have come to some sort of useful conclusion. So there's a bunch of different topics. We're either going to do an episode, maybe two or three episodes with different people on the same topic, uh, until we kind of come to some sort of consensus and then we'll move on.
Introducing Skyler and Bogdan
00:01:09
Speaker
And if that doesn't work, we'll try something different. So the first, uh, well, actually before I get into the topic,
00:01:18
Speaker
Today I have my friends, Skyler and Biden with me. We are all, uh, Laravel PHP developers at Internachi. And we are going to talk about something that came up recently. But before we get into the details, do you guys want to say hi? Hi, I'm Skyler.
00:01:41
Speaker
Hey guys, I'm Bogdan. Um, I'm here with the internet team. I currently live in South Carolina. Let's do it. Oh man. I guess, uh, yeah, I'm in St. Louis.
Inserting Data During Migrations
00:01:53
Speaker
All right. So here is the setup.
00:01:56
Speaker
This is something that I feel like comes up over and over again. And it's just like, you've got a new feature, right? So let's use this. We set out a nice little example in the docs, so I might as well look at it, right? So you're setting up a new like roles and permissions feature, right? And so we've got a migration for the roles table. We've got a migration for the permissions table.
00:02:21
Speaker
We know from the start that there are going to be a few specific roles, like there's going to be a guest, there's going to be an admin, maybe a staff, a couple other roles, right? And there's going to be like 15 different permissions. And so the question always comes up like,
00:02:39
Speaker
What do you do? Do you throw the actual insertion of data in your migration? Do you set up a cedar and run that cedar in production after you've done the migration? Do you set up an artisan command that's really just like a one-off command for inserting that specific data? Or is there some other approach? And so I was looking back even
00:03:06
Speaker
you know, over a year ago, this topic had come up and I went on Twitter and I posted a little poll.
Manual Database Insertions
00:03:14
Speaker
Oh, another option is just, hey, go into Tinker or go into your like, you know, SQL editor and just do the inserts directly. And there was a little bit of consensus, but it was kind of all over the place. And so before we kind of talk about the pros and cons, like gut reaction, which is it?
00:03:35
Speaker
Which is the best option? Is that what I kind of... Yeah, I mean, I will say like I've done the manual insert into the database and it's certainly not it. You know, there's certainly a lot of trade-offs for the speed.
00:03:50
Speaker
I feel like there's opportunities to use constants and just reference PHP strings or just strings in general that reference a database and have some sort of command that just takes some sort of class full of constants and sinks them into various places, whether that's roles or permissions.
00:04:15
Speaker
So, I feel like
SSH Discomfort and One-off Commands
00:04:17
Speaker
those are a pretty good starting point that I feel like most people reach for. What about you, Skyler? I want something as easy as PHPRs and migrate, but it always feels kind of gross to put it in the migration, especially if you're doing that migrate
00:04:40
Speaker
where you end up making a schema from your migration files, and then suddenly you've got this weird stuff in there that blows up. But, you know, bespoke commands every single time feel off too. Yeah, I mean, that's 100% where I am, is like...
00:04:57
Speaker
I hate the idea of having to SSH into one of the app servers after a migration and run a command.
Manipulating Existing Data in Migrations
00:05:06
Speaker
And then yeah, have a command that's sitting around that really doesn't do anything. It was really just a one-off thing. But at the same time, I can acknowledge that every time I've done this where I've thrown the data in the migration, I felt pretty dirty afterwards. Like I know that that,
00:05:26
Speaker
That doesn't, it doesn't feel right either. And I also feel like one of the times, I don't know if it's the last time, but one of the times this came up, the other issue was now when you run those migrations in unit tests,
00:05:45
Speaker
all this data is in there. So if you've written tests that kind of assume, like if you're, if you're doing migration based testing where you're, where you don't have some sort of, you know, world that you seed ahead of time, but you're just factoring up the data that you need for each test. Now you've run into a circumstance where you've got a bunch of records in your database that you weren't necessarily expecting to be there in your tests. And they could get in the way if you make certain assumptions.
00:06:14
Speaker
which is definitely another downside of running them in the test, in the migration rather. Yeah. I mean, I've done them, I've done things where, you know, you're, you're writing, writing tests and you are factoring up data, but then you still expect like your permissions table to have those permissions. Like even in my test, I don't, I don't want to like factory up the admin role. I just wanted to like have it there.
00:06:39
Speaker
But there are other times where you've added a new column to a table and you need to ensure some, some data is there and you don't want that in your test. So yeah, we need some solution that's going to work for, for kind of all across the different environment levels that you have.
Separation of Data Migration and Seeding
00:06:57
Speaker
Right. Well, and I mean, I guess we could take it a step back, take a step back and say, what about circumstances where you have to manipulate existing data? Right. Because I don't have much of a problem doing that in my, in migrations. Right. If I, for example, I don't know why I would do this, but if I were switching from say a, if I, if I had a given name and a family name column and I wanted to add a
00:07:27
Speaker
just switch to a name column or maybe a better example would be like, I have a given name and a family name column and I wanna add a display name column, right? I don't have much of a problem in my migration saying, update users set full name equals concat given family name. That feels okay to me. Are you guys on board with that at least or does that feel dirty in migrations too?
00:07:58
Speaker
think, like, it's not, it's not the worst. But it's still, it still doesn't feel it feels like a little bit like you're just abusing the migrations, the fact that migrations run once. And that is it feels like we're abusing that feature of migrations. It's also it also really depends on how much data there is, right? Because if we're talking about a table with
00:08:27
Speaker
100 million records in it, I'm going to approach that very differently than a table that has 5,000 records in it. Yeah, I will say, I feel like the Laravel migration system is really for doing column changes and creating tables and adding indexes. It's not really designed to be inserting data or manipulating data, in my mind at least.
00:08:53
Speaker
there's probably, I'm sure there's a way for us to do these steps. But I feel like if we take a step back and think about, well, why are we trying to automate this? Why are we trying to add rules and permissions at the very beginning of the, or at the
00:09:15
Speaker
the time where we created the roles and permissions table, we need to seed some data. I feel as if that is just an entirely different step. It's one thing to migrate data, and there's another thing to actually seed that data.
Automating Deployments and Seeding
00:09:33
Speaker
Even in the example, because you mentioned in regards to the given name and the family name to combine it,
00:09:39
Speaker
into a name, it's not really a migration. It's kind of like a data manipulation mechanism. Chances are if you were ever to run that, you probably would not be dropping a column immediately. You would run that command in the background.
00:09:56
Speaker
and have that data populate and then there's probably subsequent code that goes out or maybe it's behind a feature flag that actually takes that new manipulated data into effect. And I feel like
00:10:11
Speaker
those commands are like very specific utility commands. You don't just pipe them in. It's just over time, your data model has changed through migrations. And over time to accommodate those changes, you need to change the data. But when you're migrating a fresh set of data for testing, none of that took place. All you're doing is bringing the world
00:10:39
Speaker
up to the latest schema changes. And I feel like I have to agree with you guys, adding some sort of seeding mechanisms inside the migration system is just a very dirty trick that I feel like we've been used to because that's just the way Forge deploys or Envoy and all these other services, they just kind of run this by default.
00:11:05
Speaker
I feel like we should step back and say, well, perhaps our deployment steps should run some commands in addition to the migrations, even if they're manually configured.
Environment Setup Issues
00:11:21
Speaker
I mean, that's a good point. This, this is, this is all in the context of a like CICD strategy, right? If you're not doing continuous delivery, if you're not doing some sort of automated deployments, this is less of a concern because if your whole deployment strategy is already pretty manual doing one other manual task.
00:11:44
Speaker
you know, is not that big of a deal on the flip side. You know, at least the way we run things we do, we can do 10, 15 deployments in a given day. And some of those will be, you know, just queued as soon as the test pass, they auto merge and get deployed. And so like,
00:12:05
Speaker
Yeah, you're absolutely right. I am spoiled by having this really lovely, you know, GitHub actions and an Envoy or workflow that just makes everything happen automatically. And and I don't know, now that I'm spoiled, I don't want to go go back from that. I want some other automated process that does this piece for me. Well, I think, too, like I've come on to projects before
00:12:34
Speaker
where someone just hands me a database file and they're like, here's the local database. Or you run a command to import a local database. I've also come to projects where
00:12:49
Speaker
You just run the migrate command in Laravel, and then maybe there's a cedar, or you just have nothing, and you just have to register for an account. And then they're like, oh, yeah, the permissions table. Go into Nova locally and just add in this Google spreadsheet of permissions.
00:13:06
Speaker
And I feel like those are also things that you'd want to solve for with this approach is like, how do you, yes, in your tests, like you're testing the latest version of things, but like the everyday getting things set up or you're spinning up a new staging environment. Like, how do you get the data where you need it to be? And, and that's, I think the other use case for
00:13:32
Speaker
some sort of like idempotent data manipulation. Right. Yeah. I mean, we definitely underutilize cedars in, you know, in our project. That's not something that we really heavily lean on. And I can see, I can see, you know, a database cedar or running different database cedars being part of the solution here. But, you know, they, they kind of come, they feel,
00:14:04
Speaker
I don't know, database seeders don't feel like they're meant to be used in production in Laravel at least. Yeah. You know, and in fact, when you try to do DBC, you get a big warning if you are in prod. But you're right. A lot of the cases where you would want this in production, right? Like these are the 15, 15 permissions and five roles that are going to be in the app by default from the beginning.
00:14:32
Speaker
You're also going to often want them in your local dev environment or a staging environment or depending on the feature, something like permissions and roles that's so central to an app, you're probably going to want that in a lot of your tests.
Ideas for Data Management
00:14:48
Speaker
Whereas some other feature that is really just an ancillary feature, you probably don't want to seed.
00:14:57
Speaker
that data for all your tests because you only need it for a handful of tests in the whole suite. So it also depends on what type of data you have. It seems like there's maybe what two different types of data that we're talking about. We're talking about data that we need to be in the system.
00:15:17
Speaker
that you could run over and over again the command, and it's going to make sure that data is there. But then there's the other portion, which is massaging existing data into a new format. And those, when you're massaging data into a different format, probably don't want those to run in a test. Or it wouldn't help us, whereas running something to make sure that data is there in the database.
00:15:45
Speaker
I don't know if I'm... Yeah, I totally agree. I feel like, you know, Skyler is 100% correct in this. It's, you know, I think the case for roles and permission is quite clear because those are like very heavily dependent on like, you know, checks, like is this user a guest string or something, right? Whereas, you know, there's like these other long commands or utility commands, some may say.
00:16:12
Speaker
to actually manipulate the data. So I think what we're talking about is this initial use case of the roles and permissions. How do we make these small changes during the migration process or some sort of post-deploy process, which may not be as tricky as running some sort of long process that manipulates a lot of data.
00:16:41
Speaker
Would you agree with that? Yeah, it, it almost makes me think that we almost want something that's like kinda like migrations in that, you know, arguably once you've run your migration, you don't need it. The, that file to sit around, but it's definitely useful for it to be there. And certainly, you know, if you're going to set up another environment,
00:17:08
Speaker
you do want it to be there, but in the day-to-day operation of the application, it's kind of just like an XS legacy file, right?
New Approach for One-off Commands
00:17:17
Speaker
And in the same way, it almost feels like what we want is either one-off commands that are perhaps tied to a specific migration,
00:17:32
Speaker
or are in an entirely separate, like are represented in the database in a separately, an entirely separate table where you can run some second command that says, Hey, these are the, these are the data, data migrations or whatever that you haven't run yet. Would you like to run them? You know?
00:17:58
Speaker
Yeah, I totally agree. I feel like I kind of understand what you're getting onto. I do worry. So maybe let's step back and think about this, right? So what you're kind of suggesting perhaps in my mind is, you know, we create some sort of, you know, data migration status table of some sort, and we create
00:18:21
Speaker
files or classes that represent what needs to happen. And during deploy, we kind of say like, hey, we run the migration workflow as per usual.
00:18:34
Speaker
But then there's another command that is run right after that, right? And we would just take the files that we've defined and effectively run a database cedar in production. And then just record that that thing happened. The only thing I would, is that true, Chris? Is that kind of what you're looking for? Yeah, something like that.
00:19:01
Speaker
I can't help but think, you know, there are a lot of other commands that are sort of like one off that it might not be terrible. Like if, you know, this, this podcast is called over engineered. So I'm going to let myself go down this path of, you know, how many, how many times have we written artisan commands that are really just there for a single use? And is there some,
00:19:32
Speaker
you know, kind of migration type approach where it's like, you're not actually introducing a whole new command, but instead you're just introducing another thing that can be run by some sort of generic one-off command runner. And you could kind of do that, then you could do that for both cases, right? It's like, these are all things that need to be run once and only once, right?
00:20:02
Speaker
unless you know what you're doing, but by default, they're, they're only run once.
Dragon Code Actions Package
00:20:06
Speaker
And so you just drop some, you know, PHP file that's got sort of a date time component in the, in the, uh, file name, just like migrations do. And there's a,
00:20:22
Speaker
You know, there's like a database record called one off commands or whatever, you know, and it just scans that directory for files. It compares it against the one off commands database. And just like the migrator, it just loops through them and runs the commands one by one. And that way, you know, in the deployment story, we would just run that after migrations had been run.
00:20:49
Speaker
And I think you could even do something where you have like either a, either a use a trait or just a public, public property where you're like, like running test or is testable. So then, you know, similar to refresh database, you can have like refresh data that's going to run, but you might have a one-off thing that you don't want to have run in your tests or you have,
00:21:18
Speaker
some other thing like your permissions where you want that to run in your test suite, but like maybe by default, none of them run unless you, you know, use in tests trait or something. So one of the other options or on the same topic, right? Like I just looked in the background, how Envoy deploys data, right? The, you know, the files, everything. And, you know, we run things like, you know,
00:21:47
Speaker
Obviously, PPRs and migrate, queue, restart. And this could be effectively a second command, if I'm understanding you, Chris, that we just pipe at the end of the deployment process after the
00:22:03
Speaker
releases activated effectively, right? We just basically say, deploy this and then run this command. And in the case of the test world, you know, you may not, you know, you could choose whether or not you want to run
00:22:18
Speaker
that command similarly to how we do like, you know, database refreshes, you may choose to additionally run this data seething command in your test. And I think that's a pretty decent solution, you know, to start with.
Potential Downsides of Actions
00:22:40
Speaker
Yeah. So I actually, I was looking through this Twitter thread from, from last April or whatever, and I saw someone responded with this, uh, this package called Laravel actions, which is a little confusing because there's the, the, the, the big Laravel actions package that is, you know, for, for, you know,
00:23:06
Speaker
single actions that you can, you can execute from different contexts. But this is, this is essentially what we're talking about. So it's dragon code actions or dragon code, Laravel actions. And it looks like essentially, essentially they're doing kind of what I described, right? You have these actions that are very similar to migrations.
00:23:35
Speaker
except that they are meant to do something. Like if you look at actions, so this is how they describe it. Actions are like version control for your actions process, allowing your team to modify and share the application's actionable schema. If you've ever had to tell a teammate to manually perform any action on a production server, you've come across an issue that actions resolves.
00:24:01
Speaker
So that, I mean, that does sound like maybe someone has, has come up with a pretty good approach to this. I don't, I hate, I hate that it's called actions just because if you are using what is the other one's called Laravel actions, right? Yes. Yeah. I'm looking at the package and, you know, to be honest, you know, it kind of sounds like what we were talking about. There's effectively a database table.
00:24:32
Speaker
that represents all of the actions that have ran. And it even seems like you can roll back actions. Although I am curious, when is it intended to be ran? I haven't figured that out just yet. But this looks- Well, I would imagine it's the same thing, right? You would just put it in your deployment pipeline wherever it made sense, if you wanted to.
00:25:00
Speaker
Yeah. And they have these, like you can protect it once false or true, whether, whether it runs every time you run all your actions or, you know, execution in a specific environment takes an array or string of like which environments to, uh, to run it in or excluding from the interest. Yeah, that's neat. So you can, you could have an action that's like dev only or something like that. Yeah.
Custom Solution for Managing Actions
00:25:26
Speaker
Which is a little bit like what I was, what I was mentioning of like, how do you run this in the run this in testing, but not elsewhere. And this kind of resolves that. Yeah, this is, it's interesting. The logo is fun. Let's try again. So, okay. So, I mean, it sounds like we've come to a quick solution, but you know, what are the drawbacks to, to this approach, right? You know, I feel like, you know, is there,
00:25:57
Speaker
you know, how do we mitigate issues in production? If this runs, will this, you know, hold up deploys and so on and so forth? Right. I mean, the, the biggest concern that I can think of is, you know, what happens in the case of an action that that's, you know, operating on some million number of millions of records, right? Like if we were,
00:26:23
Speaker
I don't know, wanting to do some, some really big manipulation, you know, maybe like segmenting the audit log into like yearly, yearly segments or something like that. Right. We wanted some, some command that could take.
00:26:44
Speaker
All the audit records and generate like an audits 2022 and audits 2021. And, you know, some, I'm just trying to imagine something that's like dealing with millions and millions and millions of records. You know, in that case, yeah. Is that likely to that's, that's going to hold up deployment for sure. And I guess it kind of depends on.
00:27:11
Speaker
There are some actions that you can kind of run whenever and it doesn't matter. And there are other ones that you want to have run before the deployment finishes. Right. I mean, maybe there's some way to like mark an action as Qable. It's just like, it's a thing that has to happen once, but it doesn't have to.
00:27:29
Speaker
this package doesn't do that, but it would be, you know, that's one solution is just like, this action is Qable and it needs to run one time, but it doesn't, it doesn't affect the use of the deploy, right?
Laravel's Limitations in Migrations
00:27:44
Speaker
Versus like, well, you kind of need your permission stable set up before you can utilize. Right. Yeah. You could just, you could, I mean, you could just queue, you could just have your action be queuing a job.
00:27:58
Speaker
queue an anonymous job that's just a one-off thing or a batch of a bunch of anonymous jobs or something like that? So that's a good question. I mean, so basically you would still need to create some sort of file, right? Previously we talked about running, we would create some sort of command that you would either add it to your
00:28:19
Speaker
deployment pipeline as a thing to run after deploy, or you would have to like shell into your server and run that command. So, you know, you would still need to create a place to house more complex logic, right? So that is a benefit. You know, with this package solution here. Yeah, you would. You would basically. You would still have you would have a file for each action, right?
00:28:49
Speaker
And you would just put in your, you know, in the Envoy or config or whatever, you would put like, PHP, artisan, actions, whatever, you know, actions install or actions, yeah. Actions run or whatever, I can't remember. Although there is also kind of, it's interesting, there's a concept of before actions,
00:29:19
Speaker
Also, so you could kind of split, you could have something like in your deployment scripts, you could say like run the before actions and then run migrations and then run all other actions. That's kind of neat.
00:29:36
Speaker
Right. I do see the example that they have is basically, you know, you would do migrate and then actions before, and then, you know, you would do your additional steps and then restart your queue and then run the subsequent actions at the very end.
00:29:52
Speaker
So it kind of talks, you know, it's effectively the same thing. Um, we talked about earlier, your before actions are like database. Your after actions might be code running, running cute jobs or doing something that needs access to the whole code base. Right. With that fresh data. Right. That was just migrated thing. That thing that I always wonder about in these things is like,
00:30:20
Speaker
you don't want to end up with a situation where maybe the data exists in the database, but maybe it doesn't.
Risks of Rollback Strategies
00:30:29
Speaker
And how do you, is it up to the developer to just like write first or create right on that permission? Or is there a way to like in these
00:30:43
Speaker
I don't know, some sort of like exists method that you can write some query yourself that if it returns true, it just like doesn't, it just like bails out some sort of like validation that run that like more than, I don't know, more than just an if statement in the, in the action handler, just like some sort of validate that this should still run.
00:31:11
Speaker
Well, this, this makes me think of, this is kind of getting into episode two, where, because this is, this is a whole other class of question that is definitely related, but it's, it's a, it's a whole other topic of, you know, I need this, this record to exist. My code expects this record to exist. I need to reference this record in some way. And like, how, how do I make that happen?
00:31:37
Speaker
And I, we could jump into it now. Maybe let's hold off for a minute, minute more and see where we, where we land. But, cause that is a whole other question. And I could, I could see basically just saying like,
00:31:50
Speaker
you reserve, maybe you reserve before actions for things that need to do data manipulations that are necessary for your application, like for the next state of your application. And then you reserve other actions for any other sort of like, hey, this needs to run, but the app can run while it's happening.
00:32:17
Speaker
But, um, I mean, this, this makes me think, uh, so, you know, we were experimenting with. With a smaller side project and, and we were playing with some of the new hot technology. And so, uh, for a little while we were experimenting with like deploying to fly and using planet scale for the database and.
00:32:43
Speaker
PlanetScale basically forces this same question on you, which is, or I don't know, there's not a great story for automated, automated, automated migrations with
Perspectives on Rollback Utility
00:32:57
Speaker
PlanetScale because they have their whole sort of version of, of pull requests, but for database deployments, database migrations. And it's, it's a little bit different, but essentially,
00:33:14
Speaker
You know, their, their point is there are some migrations that you need to do before the application changes. And there are some migrations that you need to do after the application changes. Right. In some cases in, in the original case that I was giving, if I were switching from a.
00:33:33
Speaker
a first name and last name column to just a name column, I would need to migrate to add a name column. I'd need to then copy my data from the first and last name columns to my name column.
00:33:47
Speaker
And then I'd later have to migrate to drop the first and last name columns. But before I did that, I'd have to deploy a new version of the application that uses the new attributes. So it's sort of like a multi-step process. And depending on the data and schema manipulations that you're doing, sometimes they need to happen before deploy and sometimes they need to happen after.
00:34:15
Speaker
While they are 100% right that that is technically true, in my experience, 99% of the time you can just run migrations during deploy. And as long as you know that that's happening and you've architected the code correctly, it just works and you never have to think about it. And for those few cases where
00:34:40
Speaker
it does really matter. Well, then you just deploy the migration in one phase, and then you deploy the code that relies on that migration in a second phase. But I don't know, in the last several years, I can only think of maybe having to do that once or twice. It just, I don't find that it comes up a ton. And so it's kind of the same here. It's like, well, hypothetically,
00:35:08
Speaker
you're going to need to sort of orchestrate these actions and migrations and deploys and in this like really, really sophisticated balancing act. But in reality, I don't know, 90% of the time you can just migrate and then run a command that, that updates, you know, a few thousand records in a second and
00:35:37
Speaker
just live with the consequences of, oh, maybe someone happened to hit, hit the app during that second. But realistically, the way we do deploy is that's not even usually going to be an issue. So I think the question is not whether or not it's performance or when we do it. It's like, I think is the migration system, the right platform for making this change? Because yeah, you're right. You know, we could just stuff everything. You know, we could just easily.
00:36:07
Speaker
say like, hey, when this thing runs, just execute this file. I don't think it's really an issue with just deploying. I think just getting data in there, we obviously found solutions, whether it's some other utility putting into a migration file. I think it's just like, well, how can we do it efficiently? Because that's part of the challenge here in my mind.
00:36:38
Speaker
Sure. Yeah. I mean, I think I'm convinced that doing it in migrations, especially since someone has already done all the work of writing a package that does close to what I want.
Rollback Experiences
00:36:53
Speaker
I'm, I'm more easily convinced that doing it in the migrations is not necessary. You know, I, I am always a little bit hesitant to add, add a new package that, that is,
00:37:07
Speaker
sort of like a foundational decision, right? Without really thinking about it. Cause you know, if we were to go with something like this, I think, you know, we'd want to be going all in with it. So I want to know that we're going to want to stick with it for the long haul and not just like, you know, have two or three actions that we run through it and then never use it again. And now it's just another thing that slows down the deploy by a few seconds.
00:37:37
Speaker
But that said, I mean, I think the model, whether it's through this package or something that you just kind of put together yourself, the model of having whatever they are, actions or one-off commands or data migrations or whatever you wanted to call them, having them as their own thing that sort of follow the same general flow as migrations but are not migrations does feel good.
00:38:07
Speaker
Yeah. You know, I liked, I liked this package cause they talk about just invocable methods, but later on they talk about having up and down methods. I don't know how I feel about like migrating data, having a rollback strategy, like
00:38:23
Speaker
Yeah, Taylor, you just should migrate forward. I mean, I feel similarly about database migrations, but it's, uh, I know like that's a, I don't know, maybe that's too spicy of a topic. I will say, you know, I feel like, uh, I listened to a podcast with Taylor, uh, and you know, he revealed to me many years ago that
00:38:48
Speaker
the down method is optional. And part of the reason it's optional is because chances are when you're going to roll down, it's not as easy as just, here, just roll back these changes. A lot of things probably happen. It's not as simple as just roll back the migration and production. You have to move forward almost.
00:39:13
Speaker
So this, I don't know if this is an appropriate topic for this podcast, but I, I was having this argument with Aaron Francis like a few months ago and he is very staunch. Never, never, never go back proponent. And I get, I get that. I don't know. I get that thought process, but
00:39:42
Speaker
The one time, and I think it's only one time, maybe it's been two times, but the one time that PHP artisan migrate down saved my butt is worth writing the down methods in every migration to me because
00:40:05
Speaker
every once in a while. The thing is, when you are in the worst case scenario, we just deployed code that is really fundamentally flawed in some way that no one anticipated. The only thing
00:40:24
Speaker
Like you're in panic mode and you wanna just like get things back to what worked as quickly as possible. And like I said, I can't remember what the scenario was, but I know that there was one of these deploys maybe two or three years ago where we just deployed a feature that,
00:40:49
Speaker
had some fundamental problem with it. And it took the whole site down, everything was busted. And I was able to quickly roll back in Envoy, so just deploy the previous
00:41:05
Speaker
previous commit to the production branch and run PHP artisan down. And, you know, within a minute, everything was back to where it needed to be. And to me, being able to do that, just, you know, now it's just like a, it's like a
00:41:30
Speaker
security blanket for me. It's like, I know that I have it, and I do, I'll run, you know, partisan down sometimes in testing, certainly in local testing.
Naming Conflicts in Laravel Packages
00:41:45
Speaker
but I'll run it just to make sure it works, right? I'll, I'll, I'll migrate up and down to make sure that in, you know, if there's a feature that I'm maybe a little bit anxious about, like, let me just make sure that if I need to, I can get down and, and I don't know, I'm still a believer.
00:42:03
Speaker
I think, like all things, it has its place. I don't know. I've been bitten by it in that we migrated a big table, like millions of records, adding a column. In staging, it didn't have millions of records and the thing was fast.
00:42:25
Speaker
we thought like something happened. Let's roll back, but the table was locked. It rolls back, just locked even longer. And, uh, you know, like, I think it's all, you know, it's all situational, but you just ended up getting in a situation where you're like, Oh, well shoot, rolling back didn't work. Like what?
00:42:44
Speaker
what else needs to happen and, but you know, it's, I think there are times where it can be useful and there are times when it can be dangerous to roll back. Yeah. No, I can appreciate that.
00:42:59
Speaker
Yeah, I will have to second Skyler, you know, I've been in a similar situation where everything worked fine than the migration time down production. And, you know, it's like, well, we just made a bunch of code changes. Things are writing to different tables now because we've changed the migration and, you know, like adding an index timed out effectively. And it's kind of like, well, if we roll back, you know, you know, you're going to lose all of that data that has just been
00:43:28
Speaker
added by users.
00:43:30
Speaker
in these new columns. And it's a tough call to just roll back. And in my experience, I too write the down method in migrations, but it's mostly just so I can migrate down when I need to create a column change in the migrate up strategy of migrating database when I'm doing local testing. So that's kind of where I reach for the down method. I don't think I've ever
00:43:59
Speaker
Really ran in production. I think it's super dangerous. And I suppose that's where planet scale migration strategy comes into place where they run both sets at the same time and write data at both places. And yeah, I mean, that's incredible. Right. Yeah. Okay. So this, this is, this is maybe a silly thing for me to be annoyed with, but I'm, I am a little annoyed.
00:44:27
Speaker
The other, or not annoyed, but like it's a consideration. The other thing that makes me question this specific package is like, I have always thought that the, uh, the, the, the Laura's leave, uh, let whatever, I don't know how to say his last name, the, but the, the Laravel actions package that is, you know, for
00:44:57
Speaker
actions that are just a single unit of action that then can be called from controllers or called from commands or called inside of jobs, like the example of
00:45:16
Speaker
User registration is an action and you can execute it from your registration controller, but also if you want to manually register a user, you can call the same action from a command and
00:45:29
Speaker
That whole, that concept and that package has always really appealed to me. And I have always been curious about introducing it into the project that I'm working on.
Why Laravel Lacks Core Features
00:45:42
Speaker
And so there's a part of me that's just like, God, I hate the idea of having two different things called actions. Sure.
00:45:50
Speaker
Well, I think dragon code actions is maybe a stepping stone for what you're looking for, what we're looking for, but you know, a name change is easy enough. Yeah, I guess so. We can live with that.
00:46:08
Speaker
Yeah, I will say I've looked into this layer of elections package before. I mean, yes, it is compelling. And I feel like it is interesting. And I feel like both of them have their own use cases. And I understand the name conflict. Obviously, we can rename it. But at the same time, you could theoretically use, could you use this as a migration
00:46:37
Speaker
Strategy like seeding data as actions. Is that a thing or no? Which, which action are we talking about? Delirableactions.com one. Well, yeah. Could it work in the same way? Cause it does seem like it does a handful of things. It doesn't, but it doesn't have that piece where it tracks which actions have been executed in the current environment. And so you'd have to re implement that, but I suppose you could.
00:47:07
Speaker
If you just added that, if you essentially used the Laravel actions package, the Laravel actions.com package as the foundation for these action objects. And then you essentially just like, this has a whole like as object, as controller, as job. So maybe you would just have
00:47:36
Speaker
a special interface that's like as tracked action or something like that, you know? And if an action implements that interface, then it would be somehow pulled in as part of this command. I don't know. Sure. I do wonder, you know, if we step back a little bit, like why doesn't Laravel have something out of the box, right? And are we like over engineering something that should be already here?
00:48:04
Speaker
And, you know, are we just under utilizing cedars? You know, why isn't there something in the framework itself? Yeah, that's a good question. I mean, I think, I don't know, like you could use cedars, you can use database migrations. I think the thing that most people use anecdotally would be
00:48:34
Speaker
writing commands that, that changed the data when you need to. And I've seen, I've seen like, yeah, you write your migration and then you have your, you write your command and then you call the command from your, from your migration file so that it only runs once, but during the deploy, but then it's like still there to run other times, which feels, it feels gross, but it's like, well, how do I,
00:49:02
Speaker
How do I get around the 18 steps to deploy at a large company, but still get this thing to run without access to the server? Well, in the gigantic sample size of 37 people a year ago,
00:49:21
Speaker
A little more than half said they just make the changes in the migration. 30% said they were on a custom command, and the other 20% said they just update directly through Tinker, through SQL of some sort.
00:49:38
Speaker
Yeah, I definitely feel like we need a solution. Those options all sound bad. And that's what we do. I feel like there has to be almost like part of the reasons why I feel like Laravel appeals to me and even like things like rails and stuff like that is the migration system is all these features like queue systems and everything like that. And this is one of those areas where there's almost like not a clear picture of how things need to be. There's a lot of, you know,
00:50:06
Speaker
homemade solutions to this problem. And I feel like, you know, narrowing down a good solution, even if it is this actions package, you know, just to make that kind of a thing that is available as part of the liberal ecosystem. I think that would be an incredible step forward for the framework itself.
Session Reflection and Future Topics
00:50:28
Speaker
Well, I mean, I agree. And I, I happen to know that somebody
00:50:37
Speaker
who is a, who is a Laravel core contributor might be working on something here. And I'm hoping that we can, uh, we can convince him to hop on for maybe the second episode. Cause he, uh, he teased something that I, um, he, he said there may be a lower level solution to this problem that no one has suggested yet. And I've been thinking about that and I have no fricking idea.
00:51:05
Speaker
what that means. But until we get a glimpse of what that might look like, I think that this Dragon Code action seems like a really great option. I think I'm tentatively willing to say that that's my vote for how we might go about accomplishing this moving forward.
00:51:31
Speaker
Yeah. You know, in my, my hypothesis as to why it's not in the framework already is that like people don't stick around projects long enough to run into the problem of having to migrate large amounts of data. Like either, either your side project, like it's big, but maybe it's not so big that this is hard to do or like at big companies that are running Laravel, like
00:51:57
Speaker
developers leave every few years. And so different people implement this in different ways and like just nobody, nobody has the time, like no product manager and a big company is going to be like, yeah, go contribute this back to the framework and, uh, figure out like a good way to do it. They're just like, no, right. Write the SQL directly or like whatever, you know, to, to make things happen.
00:52:24
Speaker
Yeah, it's also, there's a balance. Like it's always a balancing act of like how much is appropriate to be in the framework and how much is appropriate to be in, you know, third party packages. Cause you know, there's, there's always that, uh, temptation to just add everything, add everything to Laravel and I get needing to push pretty hard against that from the maintainer perspective.
00:52:51
Speaker
Yeah, I could see that. Yeah, I don't know. It's just something that kind of ever since we started thinking about this topic, it's something that's been looming in the back of my head, like, well, why isn't there? You know, it seems like a pretty, you know, approachable solution, especially with this literal actions thing. You know, I feel like we're, I don't know. I suppose when we have the special guest on, given that he agrees, maybe we'll get a glimpse of the future ahead of us. So I'm excited.
00:53:19
Speaker
It could be, I mean, it is kind of an interesting thought experiment to say what if, what if migrations had some sort of grouping mechanism? And so you could filter down migrations by group. And then I could, in my migration class, I could have some sort of public static
00:53:47
Speaker
public static property or public static method that could be called that's like the migration group, right? And then in your application, you could run like PHP artisan migrate dash dash group equals schema, right? Or default or whatever. And then later you could do PHP artisan migrate
00:54:15
Speaker
dash dash group equals data or whatever or group equals post deploy and group equals pre deploy or you know like if you just had some mechanism for sort of defining migration groups and like by default the group is default and from the command the group is default implicitly unless you set it and that way
00:54:44
Speaker
If you define a migration that's in some other group, it just only gets run if you call the migrator for that group. That could actually solve this problem for the most part using all the same tools that we have in place right now.
00:55:05
Speaker
Yeah, almost like, almost like PHP unit test annotations, like the group grouping, your tests, your migration that, or I wonder if there's like some updates to the Cedar component itself to like, maybe Cedars can keep track of if they've been run before, or they can keep track of, so you can seed, you can see it, or you can,
00:55:33
Speaker
like seed wants. That is an interesting, that's an interesting, you know, idea like what if so-called special guest is kind of implementing this Laravel actions or has an idea of doing this Laravel actions as part of the cedars, you know, because I think like one of the big downsides of Laravel actions is there's now another, uh, set of files in, you know, perhaps the database folder.
00:56:02
Speaker
you know, alongside with migrations and cedars and now you have these actions to keep track of. And I feel like that is just like another thing to look out for and like, well, when did this run and when did that run? You know what I mean? It's, it's a bit difficult to, I'm not sure if I could answer that question because I haven't used that app, but you know, how, when do you know when something was run, right? You know, during which deploy, like is it connected to a migration?
00:56:32
Speaker
Or is it just another, you have to like match up timestamps to see, you know, during which kind of workflow this happened, given that you were creating a table and, you know, adding some records, you know, that's personally my kind of like, uh, challenge with, I suppose this actions app package. I apologize. Yeah. Yeah. It's just like a whole new thing to keep track of. Right.
00:57:00
Speaker
Perhaps a good way to solve this is to use database cedars. Or inside the migrations, there's a before hook and maybe an after hook. And during the pHPRs and migrate step, maybe there's a before thing that you can then call your little command to do whatever data manipulation.
00:57:25
Speaker
and maybe there's an after command method on there that you can just define whatever needs to happen after that thing has successfully ran. So maybe that's also an option. Yeah, or even like, I was just thinking like, what if you could just have any public function on your migrator and like artisan migrate
00:57:56
Speaker
is just implicitly calling artisan migrate up and you, you swap out artisan migrate down with something that is artisan migrate space down. Like you're giving it the first argument of down and it's just running all the down commands. That's an interesting idea. Right. And you could do artisan migrate foo and it just runs through all the migrations
00:58:25
Speaker
and looks for the function called foo and runs them. You know, I understand that there are some problems with that, but it's kind of an interesting idea. Yeah. I don't, I don't love that, but I think it's, it's definitely interesting. Yeah. I don't know.
00:58:46
Speaker
I feel like we, we've come to the, to the other side of this Hill and now we're walking down the path to worst solutions. Well, it wouldn't be over engineered if you didn't have to like, you know, start over a couple of times. That's fair. That's fair. Well, I feel like we, we literally just hit one hour and it kind of feels like this is a good place to start wrapping things up.
00:59:11
Speaker
I want to spend a little time just thinking about it. I'm only 90% convinced. And I want to see if anybody else who we've talked to about maybe coming on for an episode, if they listen to this and say, well, you didn't even talk about, I'd be curious if there's some other strategy that none of us have thought of. And that's kind of the concept, right?
00:59:41
Speaker
I'm thinking of this as a serial podcast of mini mini series, right? So each topic is sort of going to be its own season and we're hopefully going to have a couple of episodes on each topic. You know, that's going to be determined by how interesting that is. I don't, I obviously don't want to have the same conversation over and over with different people.
01:00:05
Speaker
So it's not 100% over, but I feel pretty good about where we landed. How about you two? Yeah, I like it. I, uh, this feels like what I was, what I was hoping, what we'd come to this, this agreement. So.
01:00:20
Speaker
Yeah, I think this format, I feel like the challenge that we have on our daily work schedule is we don't allocate that much time giving too much deep thought, maybe individually. But I feel like coming together and thinking about these ideas and looking at some pros and cons is a great solution. And I really hope that the topics, the packages that we looked at, the ideas that we had,
01:00:48
Speaker
are challenged by other people because I know that we will only scratch the surface on this. I can imagine 50 other mechanisms of doing the same thing. So awesome.
01:01:01
Speaker
Well, so I have a small list of things already. I know for sure, maybe it's not going to be episode or season two or however we, we structure it, but I know for sure the idea of how do you reference data from code that needs to be in the database? Like I, I definitely want to approach that.
01:01:28
Speaker
Because I have, I think I have landed on something that I like, but...
01:01:35
Speaker
I bump into this another, another friend of mine that I like hop on pairing sessions with. Like it has come up multiple times on the app that he's working on. Like, and, and we just, we deal with it all the time. So I definitely want to get there. I also feel like, um, thinking about custom relationships, custom eloquent relations is something that I am really interested in.
01:02:06
Speaker
I don't know, way, way overthinking. Is there anything else? I just want to like, uh, I suppose think about what you just said about the referencing data, um, from code in the roles and permissions world, right? After we've done all this seating somehow, uh, we want to be able to reference like what is an admin and how can we resolve that to an actual model? That's what you're referring to, right? As a role.
01:02:32
Speaker
Well, yeah, I mean, if your roles and permissions set up is truly, like truly 100% configurable, then maybe not, right? Because if you need to be able to set up any role and any permission. So roles and permissions maybe isn't a great example, but like an example that just to kind of be thinking about is like,
01:03:00
Speaker
Imagine that you are... I was looking at a video on Laricast about building a product stock tracker. If you wanted to build something that lets you keep track of who has PlayStation 5s in stock across these 10 vendors. And this is a real series on Laricast where Jeffrey Way was walking through how you would do that.
01:03:26
Speaker
And one of the pieces there is well i want say best by and amazon and you know. Some other and game stop or whatever as vendors in the database somewhere.
01:03:43
Speaker
but I'm also gonna need to specifically reference Best Buy in my code because I need like some special Best Buy crawler or something that's specific to that vendor. So it's like, I know that there is a vendor record in a vendor's table somewhere that represents Best Buy.
01:04:04
Speaker
And I need to load that up so that I can create new records that are associated with that database model. And so like, am I just going to hard code? Like this is the ID in the database for Best Buy? Or am I going to just look up?
01:04:25
Speaker
the vendor whose name is best buy and hope that, that, that vendors name or, you know, the way I, I, I wrote the name never changes. I never need to go back in and add like a registered trademark to it or, you know, right. Okay. Something like that. That, I think that's a better example. Yeah. No, that, that definitely is a challenge I think for many people. Awesome. Yeah.
01:04:54
Speaker
Are there any other things that we've talked about that kind of fall in that category that we can just kind of weave as stuff for us to think about or for anyone to kind of ponder as they're an hour and six minutes into this podcast if anyone ever gets an hour and six minutes into this podcast?
01:05:18
Speaker
I don't know. I mean, I'm just, I'm curious what other people, uh, what other people think or feel or like what things they wish they could over engineer. Yeah. 100%. Everyone has had this experience of like, God, I D I have to do this all the time. And I wish there was a better way, but I've never had an excuse to actually figure out the best way.
01:05:46
Speaker
All right, well, with that, I guess I'm going to click this button and we're going to stop. Sound good? Sounds great. Sounds good. See you guys next time. All right. Thank you, guys.