This past summer, Chef Tory Miller opened the doors to his fourth restaurant, Estrellón, right in the heart of Badger Country. Given his reputation among Madison’s food faithful, it was hardly surprising to see the small-plates concept, which translates to “star” in Spanish, find its footing in the capital city of America’s Dairyland. The real revelation came shortly after.

The winner of the 2012 James Beard Best Chef: Midwest decided his most recent restaurant would also be his last. While it’s rare to see a culinary empire step on the brakes so abruptly, Miller isn’t anywhere near lacing up his chef shoes for the last time. If anything, the reality is just the opposite.

“I think as a chef you always have that entrepreneurial spirit. You want to keep pushing, keep driving to expand, grow, and do something different. But there is a certain point where you have to sit back and be thankful for what you have, and try to be the best you can be in all of your concepts,” says Miller, who was one of 20 semifinalists for this year’s James Beard Outstanding Chef.

In Chef Miller’s world, the idea of scaling back remains a relative one. He announced plans last month to oversee the menu at upcoming Dane County Regional Airport concept, Mad Town Gastropub. He recently turned 40 and has two sons—6-year-old Remy and 2-year-old Miles. He’s cooking seven days a week and still chasing that mythical 10-hour shift. “That is a real [goal],” Chef Miller jokes. But the crux of his decision came down to simple priorities. In addition to spending more time with his family, Chef Miller wanted to return to his roots. Literally. Managing his four restaurants—L’Etoile, Graze, Sujeo, and Estrellón—was vexing too many aspects of his entrepreneurial spirit. Chef Miller wanted to laser in on the food, local growers, and the connection with diners that had shifted his career from the bright Big Apple lights to Madison in the first place.

After graduating from the French Culinary Institute in New York City (now the International Culinary Center), Chef Miller worked gigs in high-profile kitchens like Eleven Madison Park and Judson Grill before deciding to return to the Midwest. Born in South Korea, he was adopted by a family in Racine, Wisconsin, when he was 18 months old, and grew up on a farm and in a restaurant—the still operating Park Inn Diner. One Wisconsin trait that stuck with him, even as he was navigating the ruthless chef culture in New York City, was his home state’s dedication to its food sources. According to the Wisconsin Milk Marketing Board, there are 9,900 licensed dairy farms in the state (as of 2016) and 96 percent of them are family-owned. “It’s still shocking to me when restaurants in Madison are talking about where they get their food,” Chef Miller says. “I believe sourcing should be a way of life and not a way of marketing.”

Despite his Wisconsin ties, Chef Miller was unfamiliar with Madison when he arrived in 2003. One thing he did know: When it came to the state’s culinary landscape, it simply didn’t get any bigger than Chef Odessa Piper, the winner of the 2001 James Beard Best Chef: Midwest. Her restaurant, the French-focused L’Etoile, opened in 1976 and, in addition to becoming a city landmark, was a driving force in the growth of the Dane County Farmers’ Market.

In 2005, Miller purchased the restaurant with his sister and business partner, Traci Miller, after spending two years as Piper’s chef de cuisine. For a breakthrough solo venture, this was both an amazing and terrifying place to start.

“The best thing was that I had two years of experience in the kitchen at L’Etoile, working as a chef, before we bought it. But in the public eye it was, ‘Who is this new chef cooking at L’Etoile?’ I had been there for two years. If you had eaten there, you had eaten what I was making. So mentally I was ready,” Chef Miller says. Still, trying to face down a legacy was intimidating. “I was always chasing [success] in the beginning, and I feel like it was pretty detrimental to my personal relationships and my relationships with my cooks,” he explains. “I eventually realized what I was doing and that I didn’t need to outdo anything or outshine anybody. I just needed to be myself. That kind of shift made a big difference and made me a lot happier.”

[pagebreak]

Chef Miller moved the restaurant to the U.S. Bank building on South Pinckney Street in 2010, allowing for his second concept, the gastropub Graze, to open next door. The new location also increased capacity by nearly 100 percent and included a large private-dining area. Additionally, the move helped Chef Miller truly imprint his mark on the time-honored restaurant. The validation came in resounding form when Chef Miller literally knocked someone over rushing to the stage at the 2012 James Beard awards.

“I still can’t believe [Chef Dan Barber of Blue Hill] said my name. I actually had to go back and watch the video to live it. When it’s happening, you just run up there and say a bunch of gibberish,” says Chef Miller, who crashed into famed chef Jacques Pépin on his way to the podium. “I had to grab [Chef Pépin]. I was like, ‘Oh, my god, Chef, I’m so sorry.’ He was one of the deans at my culinary school. He was like, ‘It’s OK, Tory, go get your award.’”

Industry recognition and the diversity of his Deja Food Restaurant Group, has led Chef Miller to the realization that this might just be a perfect place to start living in the now. “After the first season, and meeting all of the farmers and seeing the community, I knew I wanted to be committed to Madison. But the idea of being where we are now? It never really occurred to me. Ever,” he says. “And it’s still one of those things where I’ll be sitting in one of the restaurants eating with my family and look around, see people being happy and the staff doing a great job, and just realize that we built this, and I was a part of it. It’s just incredible to think about.”

Tory Miller: Rumor has it that Estrellón will be the last concept you open.

That’s true, and it’s really just the restaurant culture right now. I feel like, especially for cities like Madison, there are a limited amount of customers. There’s a limited amount of people. I think as a chef you always have that entrepreneurial spirit. You want to keep pushing, keep driving to expand, grow, and do something different. But there is a certain point where you have to really sit back and be thankful for what you have, and try to be the best you can be in all of your concepts. I do use four very different parts of my brain to express what I want to do in all of those restaurants. It’s gotten to the point where I think, ‘OK, there are a lot of restaurants in Madison and we have four pretty great ones.’ We just want to continue to focus on what we’re doing.

Did it feel like you needed to slow down?

I think a lot of it has to do with how I feel right now about opening more restaurants. Having done Sujeo in 2014 and Estrellón in 2015, and really, Graze opened in 2010, and L’Etoile in its newest location opened at the same time. It’s been a really hectic six years. I’ve been managing a lot of things. It’s not just the amount of customers that can be in our restaurants every day, it’s also the constant juggle of finding enough staff, making sure everyone is happy. If you have almost 200 employees between four restaurants it gets to be very, very hard to fill all those positions. So, I think, for me, doing Estrellón was really tiring and kind of made me want to just get back to cooking more than managing.

You turned 40 this year, and have two young sons, 6-year-old Remy and 2-year-old Miles. How much of this decision-making process came down to finally striking a reasonable work-life balance?

Every time you do anything that takes you away from your family, that thing has to be really, really worth it. What we have right now is basically pushing the limit of how much I can give of myself outside of my family. As my kids get older, I want to be more involved and do all of the daddy stuff, and the husband stuff. That’s important to me.

How are you at delegating?

I’m getting there. I definitely cook in all of the restaurants every day. Depending on the day, it depends on where I am and what I’m focused on. I’ve gotten a lot better at delegation and being able to let people make mistakes. I try to follow the model of making everyone better, and you can’t make people better by just doing everything for them. I want all of my chefs to be bigger and better than I am. The only way for them to get there is to do it on their own. I can come in and look at the dish and let them know when I don’t like what I see. I’ll tell them that. But for the most part, when you’re surrounded by a super talented group of chefs who are in the kitchens every day, and execute things the way I want them to, and are strong enough to make their own decisions, it’s refreshing. For me now, cooking is a passion more than anything. I get to have this awesome staff and execute the menu, and I can come in and make things that I want to make; put dishes on that I want to put on, and collaborate. I just turned 40. That helps keep me young and pushing to cook better food every day.

[pagebreak]

If one of your employees was describing you to a complete stranger, what do you think they would say?

That’s a good question. I think people have a little bit of a fear factor when it comes to letting me down. I used to be able to tell people feared me because they thought I would scream or throw stuff at them. But I decided a number of years ago that I wasn’t going to be the cartoon chef anymore. It was a difficult transition because coming up in New York in the late ’90s and early 2000s, it was a scream fest. It felt like you were going to murdered by the chefs. And part of that rubbed off. It rubs off on people in different ways. Some people say, ‘I’m never going to do that.’ And other people say they’re never going to do it but once they’re in that position, it just comes out. For me, I’m leaning more toward getting the best out of people through gentle nudging. I’m definitely not a big yeller anymore. I just try to inspire people through what I do and the message I’m trying to deliver, which is to work hard and not accept mediocrity, and to understand that it’s a gift guests are in our restaurant every day.

Is it humbling to think about what you’ve accomplished in Madison and how it’s affecting the future of the city’s restaurant culture?

Totally. It’s pretty awesome to see people who have come through our kitchens or came on the scene after I did. I’m really impressed by what they’re doing in their kitchens and to see where they came from—literally cooking out of cookbooks to being inventive and pushing a culinary movement. I think it’s amazing. It’s also just timing. I feel like cooking and food as a political statement is really big right now. You can see now, with a lot of chefs, that their philosophy matches their culinary style. I think that’s really rad. You see a lot of smaller restaurants with smaller menus, but intricate, delicious, well-thought, well-sourced menus. Those kinds of places couldn’t survive in New York, Chicago, or L.A.

Making the bold move from New York City to Madison, has Wisconsin lived up to everything you hoped it would be?

It’s taken a long time and it’s still shocking to me when restaurants in Madison are talking about where they get their food from. I believe sourcing should be a way of life and not a way of marketing. And I think it’s odd when people do that. I worked in New York City with the great city market, and then going to Odessa [Piper], who obviously helped build the Dane County Farmers’ Market here, it’s been all about sourcing. It’s been years and years of doing things one way. It becomes normal and natural. So if people are like, ‘Oh, man, I really want an avocado,’ it feels weird to me because we don’t have them here. The culinary scene in Madison is finally really catching on. At its core, and from the day I got to Madison, it’s been about the farmers’ market. It’s such a big part of life here, and that’s really great. Madison used to shut down really early. It’s kind of grown up a little bit.

You’ve been using a put-by system since day one, where you store and freeze certain ingredients so they’re available year round. How has that changed now that you own four restaurants?

It’s super intense. This year, we actually played it perfectly with the rhubarb. We pulled the last bag of rhubarb out of the freezer for last season. Then, on the Saturday of that week, we got rhubarb at market. It was down to the wire for fruit. This year, for ramps for example, we were buying, instead of 20 pounds at the time, we were buying 80 pounds; pickling and making kimchi. That sort of stuff—drying and freezing the greens. It becomes natural. I always tell everyone, the way we’ve built that system, with freezing and staffing—you need people who just process food for the winter—not a lot of restaurants can do that, spatially or staff wise. Restaurants are built on convenience and being able to get ingredients all year round in case you don’t want to change your menu. If you do change your menu often, you’re probably too small or you don’t have the storage capacity.

How is this easier in a city like Madison than somewhere much larger, like New York?

One of the great things about the community in Madison is that we’re able to talk to people, like root vegetable and potato growers, in the fall. And some fruit growers will freeze fruit for us and keep it where they are. Apples and cherries are big ones that we get. In the fall, we’ll say, ‘Hey, commit us to 2,500 pounds of cherries’ and they’ll save them for us. We’ll get them seven buckets at a time. It’s the same thing for the root vegetables. It’s really nice to be able to count on there being turnips and sweet potatoes and sun chokes and celery root throughout the winter, and not needing the facility to store that. Everything we do is about the relationship with the grower, and we’ve spent four years cultivating that. I think that alone enables us to say, ‘Hey, you guys know that we’re going to take this stuff throughout the winter. Save it for us, and we’ll make sure that it doesn’t go to waste.’ A lot of restaurants, if you only go about things in a marketing way, farmers aren’t going to be into that. It’s not going to be a lifelong relationship, with the give and take that you need.

When you arrived in Madison, did you ever think you would achieve this level of personal success?

When I first came to Madison, I really didn’t know what to expect. I had never lived here. I had heard of Odessa just briefly because she won the Beard award in 2001, but I really didn’t understand the scope of what Madison was in terms of the farming community and how closely rooted a restaurant could be to the surrounding farms. I wasn’t thinking much about my future. I needed a job and I wanted to work at the best place I could. After the first season meeting all of the farmers and seeing the community, I knew I wanted to be committed to Madison and the city. But the idea of being where we are now? It never really occurred to me. Ever. And it’s still kind of one of those things where I’ll be sitting in one of the restaurants eating with my family and look around, see people being happy and the staff doing a great job, and just realize that we built this, and I was a part of it. It’s just incredible to think about.

[pagebreak]

What were those early days like?

When I was just a chef at L’Etoile, I never thought I would make it there, but it all just started clicking. Once I started getting the food from the farmers and started connecting with the food as an extension of them—where I understood how much work they did before it even got to me—that really changed how I was doing things and changed how I viewed culinary experiences as a whole.

I would never have anticipated being where we are now, even after we bought the restaurant. I feel like that first year people started whispering about Beard awards and I was just like, ‘You guys are insane.’ Then two years later, after we opened Graze, to get nominated, it was such an incredible amount of stuff in such a short amount of time. It thrills me to think about it. I often don’t even try. Chefs are weird that way. We live for the compliments but at the same time we kind of shun away from them. I think you definitely set out to always do a good job, and if you’re lucky you can continue to grow, and that’s kind of where we are now. We just want to focus on the awesome things that we have and really make them great, as opposed to expanding further.

Was it intimidating that L’Etoile, a restaurant with that much history, was the first concept you owned?

Yeah, man. It really, really was. The best thing was that I had two years of experience being in the kitchen at L’Etoile, working as a chef, before we bought it. But in the public eye it was, ‘Who is this new guy? Who is this new chef cooking at L’Etoile?’ I had been there for two years. If you had eaten there, you had eaten what I was making. So mentally I was really.

How did that challenge drive you?

There’s always that behind-the-scenes chip on your shoulder where you always want to outdo yourself. I wanted to be better than the me from last year. I was always chasing in the beginning, and I feel like it was pretty detrimental to my personal relationships and my relationships with my cooks. I eventually realized what I was doing and that I didn’t need to outdo anything or outshine anybody. I just needed to be myself. That kind of shift made a big difference and made me a lot of happier.

How many hours were you working?

Back then it was 16-hour, 17-hour days. I’ve had some times in the past where I was working 20-hour days. That was a pretty intense time.

Now?

I try to work an 11-hour day. That’s a pretty good day. We always joke about the 10-hour day being the perfect day. That is real. It’s still seven days a week for me. You try to get one off to be with your family. I definitely scaled back. Some days are longer than others. Saturdays you always start at 6 in the morning and you always end at 11 [P.M.]

What led to your decision to move L’Etoile in 2010?

It was mostly the need to make our business more sustainable. We needed to be able to expand in order to make enough money to pay the bills. We wanted to continue that relationship we had with our farmers. Opening Graze exponentially changed the amount of money we could put back into our economy. To be able to have a real sense of the difference you’re making is amazing. If a family raises our pigs and we give them money, and you see their business and farm and family grow, that’s a real thing. We wanted to continue to be able to do that for a long time. So we said, ‘Let’s open a new place that people can come to every day and still get really great, well-sourced, well-prepared food.’

What led to your next moves?

After moving L'Etoile, everything with Sujeo and Estrellón has been a factor of having the right space offered to us. We were almost recruited in a way to open these restaurants. The buildings have always worked with us on getting open. For me as a chef, your creativity never really turns off. So if someone is like, ‘Hey, this space is available. What do you think?’ OK, Asian restaurant or Spanish restaurant—that would be awesome. Then it really is kind of like a snowball effect, and the rest is history.

You don’t often see two chefs win James Beard awards for the same restaurant, at different times. Did that make it feel Tory Miller had finally put his stamp on L’Etoile?

It is the one pinnacle award or recognition as a chef that gives you validation of your entire career. It’s not like the “Best Ofs” that come out every year. This is the culinary pinnacle of our industry. You look up and see the names of the chefs that are recognized, and they’re culinary gods to a lot of us. Still, I looked at the names that are nominated each year and they’re just people that I admire so much. To be in that class, it was shocking. It was also validating, and helped my shoulders relax a little bit. I could say, ‘You know, Odessa did amazing things and I took the baton and I’m going to do alright, too.’

Really, it was a lot of personal work. Chefs, like I said, we toot our own horns but we don’t want to do it. We want to let someone else do it. It’s unbelievable anytime you get that close to a James Beard. To actually win one? It’s just incredible.

[pagebreak]

Were you surprised to hear your name called?

That was a really sketchy day. You don’t really think about what’s going to happen if I win, or what’s going to happen if I don’t win. It’s all of those emotions going into it. I remember they did the best of the Midwest early in the show. Being friends with [chef and co-owner of New York City’s Blue Hill and Blue Hill at Stone Barns] Dan Barber and have him present to me was really special. But I still can’t believe he said my name. I actually had to go back and watch the video to live it. When it’s happening, you just run up there and say a bunch of gibberish.

Did you have a speech prepared?

No way, man. It was pretty unbelievable. I definitely felt like some of the other chefs were going to win, mostly because [New Orleans restaurateur] John Besh, the night before, we got a little tipsy and he was like, ‘They’ll never give it to you the first couple of years. Just party and enjoy it.’ I was like, ‘Yeah, that’s so awesome, John Besh.’ Then, of course, I’m all hung over. It was pretty funny.

Then I ran over [renowned chef] Jacques Pépin on my way to get the award. So that was horrible. There were all these people, it was still early in the show and they were coming in and getting into their seats. And there were some really tall women. I remember trying to weave around them and I came around one of them and ran into this dude, and it was Jacques Pépin, and I knocked him over. I had to grab him. I was like, ‘Oh, my god chef, I’m so sorry.’ He was one of the deans at my culinary school. He was like, ‘It’s OK Tory, go get your award.’ I felt terrible. I just ran up on stage. It was all sorts of surreal. I remember looking down and Thomas Keller was in the first row. It was crazy.

These days you’re getting nominated for even bigger things, like the Outstanding Chef award.

I couldn’t believe it. That one was a real, real shocker. And probably as close as I’m ever going to get to winning if I’m being truthful. I didn’t expect it. Nothing is guaranteed. You don’t get to go back there just because you were previously nominated. Typically, when people make the long list you don’t want to put out there that you were just nominated, but when I saw that I was in this group, I was floored. Really, really floored.

Does it still surprise you to be mentioned in the same conversation with these chefs?

I went to my culinary school [French Culinary Institute in New York City] and they have this wall of distinguished graduates and my picture is right next to [Philadelphia restaurateur] Marc Vetri’s. I took a picture of it. This was years ago. I was like, ‘Holy shit, I’m right next to Vetri.’ Seeing him and talking to him, he’s a great, great guy. Sean Brock, too. Those two guys I know pretty well. But to be put in that same category is crazy. And to have 20 chefs in the entire country nominated, it’s really, really insane. It is kind of validating. I do work a lot. I take myself away from my family a lot to do what I do. It makes it worth it when you get recognized by your industry.

Did you intend to open four completely different restaurants?

It’s incredibly exciting and difficult doing so. We’re also trying to be smart about what we’re doing, and thinking about what’s in Madison and what isn’t in Madison. There wasn’t a locally sourced pan Asian restaurant. When I went out to dinner with my family, frankly I wanted to know what we were eating. I wanted to eat pho or ramen or Korean food, and when you knew that the stuff was coming off the back of a semi truck, you’re like, ‘Well, I’ll eat this because I really want it.’ But my conscious was telling me not to feed this to my kids. It’s a conscience decision to put food in your body and your kid’s body. I do that all day. It’s weird to do that in my culinary life and then have a Monday off and go eat garbage. It just doesn’t make sense.

The same thing with Estrellón. There was definitely no Spanish, kind of small plate, tapas restaurant in Madison. At the time, there were a couple that were trying to do tapas, but more in the name of tapas and less in the name of actual Spanish inspiration. I wanted to bring that to Madison and have another outlet. It’s also just culture. With me being Korean, it was important to make Sujeo, and I love it so much. Estrellón is an extension of a cuisine I’m really passionate about from a philosophy standpoint—ingredient forward, seasonal, and fresh.

Talk a little bit about your background.

I got adopted when I was 18 months old. I was here after that. I was really young. I haven’t gone back. It’s one of those things where there’s always been something going on that’s prohibited it. I think that time is coming very soon, though.

I’ve traveled to Spain. It was a little less daunting for me, less personally daunting. It would mean a lot to go back and see the orphanage and all that kind of stuff. But going to Spain was like killing two birds with one stone. I got to have a honeymoon with my wife and also do research and development and be inspired by what’s happening there.

How did you get into cooking?

I grew up on a farm and in a restaurant. My adopted mother’s parents owned a restaurant diner in Racine called The Park Inn Diner. It’s still there. We still go there every Christmas. We eat cheeseburgers. They owned the diner and all the kids worked there since we were young. Now, I’m doing something a little different, but similar, you could say.