F ROM “R ESEARCH ” T O “I NDUSTRY ” M OBILE EDITION (Or, How I Learned To Stop Worrying about Papers and Start Building Smartphone Apps) Stephen Miller, Co-Founder and SVP Engineering Fyusion, Inc
T HIS TALK IS ABOUT … • How to future-proof your research code • How to transition from academia to industry • Hard things I’ve learned growing from a team of ~4 to a team of >50 • A few tricks for porting desktop code to mobile • Just enough shameless company promotion to justify reimbursing the fl ight
T HIS TALK IS NOT ABOUT … • In-depth technical details • Any particular library • Science • Computer Vision, really, except insofar as it informs certain challenges
Y OU ARE …?
R OUGH OUTLINE 1: Context (~5 min) • Who I am • What I did before Fyusion • What Fyusion is all about 2: Growing pains (~10 min) • Problems with going from research -> industry • Particular problems with going from desktop -> mobile 3: Helpful tips (~10 min) • General code organization • Device-speci fi c vs generic code • Broader startup tips (git fl ow, process, management)
Robotics 3D Vision W HO AM I? My current face Open Source / Industry Collaboration
R OBOTICS @ B ERKELEY (2007 - 2011) https://www.youtube.com/watch?v=5FGVgMsiv1s
G AP BETWEEN ACADEMIC + PUBLIC EXPECTATION http://abcnews.go.com/Politics/oklahoma-sen-tom-coburn-report-shows-taxpayer-money/story?id=13689403
3D V ISION @ S TANFORD (2011 - 2014-ish) Sensor 0 • Started PhD on 3D perception • Use only low-cost ($100 or less) sensors Sensor 1
M AINTAINER @ PCL (2011 - present…ish) •World’s largest 3D Image Processing initiative 500 s r e p 250 o l e v e D 0 3/1/11 9/1/11 3/1/12 9/1/12 MONTH TO MONTH CUMULATIVE
P ROBLEM : C ONSUMERS DON ’ T CARE ABOUT TECH DEMOS Technologists Consumers Point clouds, Meshes, Octrees, … Stop wasting tax dollars and give me all the stuff I saw in the movies
P ROBLEM : C ONSUMERS DON ’ T CARE ABOUT TECH DEMOS Technologists Consumers Slick, seamless, “magic” Point clouds, Meshes, Octrees, …
T HE CHALLENGE : MAKE IT SO STURDY THEY DON ’ T KNOW IT ’ S RESEARCH
http://fyusion.com
U NDER THE HOOD : SCIENCE
T O THE USER : MAGIC
F ASHION , E COMMERCE
C USTOM CAPTURE MODES
E XAMPLE I NDUSTRY : A UTOMOTIVE
E XAMPLE I NDUSTRY : A UTOMOTIVE
H OW DID WE SCALE ?
R EQUIRED BUSTING A LOT OF MYTHS
M YTH : D EMO C ODE ™
R ECENT EXAMPLE : BOTTLENECK foreach slice [start, end]: fire up decoder march to start fire up processor process to end Slices were *nonoverlapping* and *consecutive*. This code is extremely wasteful^
GIT BLAME
O RIGINAL CODE “Just hack it!” fire up decoder fire up processor process
O RIGINAL CODE ++ “Just hack it!” foreach slice [start, end]: fire up decoder march to start fire up processor process to end Bad code evolves over time
“J UST H ACK IT ” - I NDUSTRY E DITION
T HE P OINT : DO IT WELL ENOUGH THE FIRST TIME , BECAUSE …
M YTH : T HE E NGINEERS ™ WILL DO IT LATER
“T HE E NGINEER ” WILL •Understand my hacky pseudocode • Convert my O(n^3) algorithm into O(n) with Optimization™ • Add helpful comments • Rename obscure variables and functions •Write unit tests for everything “The QArchitician is half QA, half Software Architect, half Mathematician!”
T HE A CTUAL E NGINEER WILL • Be deeply frustrated to work with brittle code as a starting point • Be very conservative about accidentally breaking something • Require your input and review time, and in all likelihood… • Quarantine your code before they fi x it
AND THEY ’ LL BE * RIGHT * TO QUARANTINE , BECAUSE …
T OO M ANY C OOKS S POIL THE A LGORITHM % you know o ff hand: 100 Old, hacky research code
T OO M ANY C OOKS S POIL THE A LGORITHM % shared: % they know o ff hand: ~60 % you know o ff hand: ~60 10 New, “clean”, “optimized”, “refactored” code
W HEN CRASHES ATTACK
W HEN CRASHES ATTACK “This looks nothing like my algorithm.”
W HEN CRASHES ATTACK “Don’t ask me, it’s not my algorithm.”
T AKEAWAY : SOMEONE SHOULD FULLY UNDERSTAND A GIVEN CODE BLOCK . I F IT ’ S COMPUTER VISION RELATED , THAT SOMEONE IS PROBABLY YOU .
M YTH : THE PLATFORM IS IRRELEVANT
There Will Be Bugs
2016 A NDROID R ECORDING B UG • Scattered reports from users on a speci fi c device by a European manufacturer: “Suddenly, the camera just stops sensing motion” • Di ff erence in timestamps: 13107153 13107234 13107212 13107128 • ^ 2^17 second o ff set
2017 I P HONE 8 AND X R ECORDING B UG • “All of a sudden it just stops working” - iPhone 8, 8+, and X users • Bonus points for anyone who can tell me what is mathematically interesting about 768614.395 seconds…
2015 V IDEO D ECODER B UG • “It works fi ne for the fi rst 204…then stops forever”
2015 V IDEO D ECODER B UG • Bug in Apple video decoder (lasted at least 3 years) video200.mp4 Decoder1 video200.mp4 Decoder1 Decoder1 video200.mp4 Decoder1 Decoder204 video204.mp4
2015 V IDEO D ECODER B UG • Bug in Apple video decoder (lasted at least 3 years) Decoder1 video200.mp4 Decoder1 video200.mp4 Decoder1 Decoder1 video200.mp4 video_sym.mp4 Decoder204 video204.mp4
E XISTENTIAL Q UESTION : H OW D O Y OU T EST F OR T HIS $@*(&? • Not scalable to actually test for every possible case “Okay, so before a PR • Irresponsible not to test for can be merged, we need to make every possible case sure we run it on this particular model of Alcatel phone for at least 3 weeks, and also the plus version, and the international version, and a version running Lollipop, and also bring it to that garage, and also…”
M YTH : I CAN JUST DO IT ON T HE C LOUD ™ INSTEAD
G OING TO THE CLOUD • 1 minute of 1080p capture on an iPhone = 200MB Strong AI • In many cities, you’ll be lucky if that is done in 5 Achieved! minutes — and that assumes Apple doesn’t kill you fi rst Just point at anything and our patented algorithm will do • If you want to scale globally (China, India), you can’t the rest assume an LTE connection
M YTH : I CAN JUST SAVE RAW DATA AND DO IT OFFLINE
D OING IT OFFLINE • 1 minute of 1080p capture = 3600 frames • If you have a simple thing (e.g. 10 ms or less), you’re still looking at 30 seconds to run it • And that’s ignoring h264 decoding time! • If you can do anything online, do it! Attention spans are very low, even for cutting edge tech.
M YTH : I T ’ LL BE LIKE A LAB , WE CAN WING IT !
I N THE BEGINNING … You and a handful of colleagues know every line of code. You move extremely quickly, and have no need for QA or code review. “Meetings” make you think of Dilbert
E VENTUALLY 50+ people, multiple products, multiple deadlines, (and this is just one repository) actual customers No way for one person to keep track of everything
R EALIZATION : WE NEED A MORE FORMAL PROCESS
M YTH : P ROCESS ™ WILL SOLVE IT
P ROCESS : A P RIMER Step 1: Commit to an insane deadline Five days? I’ll do it in one!
P ROCESS : A P RIMER Step 1: Commit to an insane deadline Step 2: Fail
P ROCESS : A P RIMER “Did you even read The Lean Startup?!!” Step 1: Commit to an insane deadline “We need JIRA!” Step 2: Fail “Trello or die!” Step 3: Blame Process™ “Scrum you fools! It’s been scrum the whole time!”
T HE TRUTH Process is useful, and necessary. But it isn’t a magic bullet.
E NOUGH MYTHS ; WHAT SHOULD I DO ?
T IP : BEWARE OF CVPR SYNDROME • “What if it also used GANs to predict when the user wants to stop recording?” • “Surely superresolution can help this” • “Why are we using JPEG? This latest compression algorithm is *way* better” • “Couldn’t a neural network handle all the on-screen rendering too?” • “This would be much better if I wrapped it in Haskell”
T IP : BEWARE OF CVPR SYNDROME https://www.youtube.com/watch?v=evUWersr7pc
I T WILL GET MORE COMPLICATED ON ITS OWN . D ON ’ T PUSH IT . “B UT WAIT , THERE ’ S …” • Customization per customer • Exceptions for certain lighting conditions • Optimizations for particular phones • New battery constraints that force you to fork and simplify • Handling of about 10000 di ff erent edge cases
Recommend
More recommend
