Skip to content

Back out "[presto][diff_train] Add a native type manager" #24961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

woshizhouyihao
Copy link
Contributor

Summary:
Original commit changeset: 66c168d92307

Original Phabricator Diff: D71918754

Reviewed By: shangm2

Differential Revision: D73492432

Summary:
Original commit changeset: 66c168d92307

Original Phabricator Diff: D71918754

Reviewed By: shangm2

Differential Revision: D73492432
@facebook-github-bot
Copy link
Collaborator

This pull request was exported from Phabricator. Differential Revision: D73492432

@pdabre12
Copy link
Contributor

@woshizhouyihao What's the reason for this revert?

@woshizhouyihao
Copy link
Contributor Author

@woshizhouyihao What's the reason for this revert?

We are seeing some errors internally as following, which are blocking some pipelines.
Reverting those 2 commits resolve the issue.
3532782
68cdacd

at com.google.common.base.Preconditions.checkArgument(Preconditions.java:218)
	at com.facebook.presto.metadata.FunctionAndTypeManager.getUserDefinedType(FunctionAndTypeManager.java:761)
	at com.facebook.presto.metadata.FunctionAndTypeManager.getType(FunctionAndTypeManager.java:378)
	at com.facebook.presto.metadata.FunctionAndTypeManager$1.getType(FunctionAndTypeManager.java:200)
	at com.facebook.presto.sql.analyzer.ExpressionTreeUtils.tryResolveEnumLiteralType(ExpressionTreeUtils.java:149)
	at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:536)
	at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:389)
java.lang.IllegalArgumentException: QualifiedObjectName should have exactly 3 parts
	at com.facebook.presto.common.QualifiedObjectName.valueOf(QualifiedObjectName.java:45)
	at com.facebook.presto.common.type.TypeSignatureBase.of(TypeSignatureBase.java:38)
	at com.facebook.presto.common.type.TypeSignature.<init>(TypeSignature.java:101)
	at com.facebook.presto.common.type.TypeSignature.<init>(TypeSignature.java:96)
	at com.facebook.presto.common.type.TypeSignature.parseTypeSignature(TypeSignature.java:216)
	at com.facebook.presto.common.type.TypeSignature.parseTypeSignature(TypeSignature.java:195)
	at com.facebook.presto.sql.analyzer.ExpressionTreeUtils.tryResolveEnumLiteralType(ExpressionTreeUtils.java:149)
	at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:536)
	at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:389)

@pdabre12
Copy link
Contributor

@woshizhouyihao What's the reason for this revert?

We are seeing some errors internally as following, which are blocking some pipelines. Reverting those 2 commits resolve the issue. 3532782 68cdacd

at com.google.common.base.Preconditions.checkArgument(Preconditions.java:218)
	at com.facebook.presto.metadata.FunctionAndTypeManager.getUserDefinedType(FunctionAndTypeManager.java:761)
	at com.facebook.presto.metadata.FunctionAndTypeManager.getType(FunctionAndTypeManager.java:378)
	at com.facebook.presto.metadata.FunctionAndTypeManager$1.getType(FunctionAndTypeManager.java:200)
	at com.facebook.presto.sql.analyzer.ExpressionTreeUtils.tryResolveEnumLiteralType(ExpressionTreeUtils.java:149)
	at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:536)
	at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:389)
java.lang.IllegalArgumentException: QualifiedObjectName should have exactly 3 parts
	at com.facebook.presto.common.QualifiedObjectName.valueOf(QualifiedObjectName.java:45)
	at com.facebook.presto.common.type.TypeSignatureBase.of(TypeSignatureBase.java:38)
	at com.facebook.presto.common.type.TypeSignature.<init>(TypeSignature.java:101)
	at com.facebook.presto.common.type.TypeSignature.<init>(TypeSignature.java:96)
	at com.facebook.presto.common.type.TypeSignature.parseTypeSignature(TypeSignature.java:216)
	at com.facebook.presto.common.type.TypeSignature.parseTypeSignature(TypeSignature.java:195)
	at com.facebook.presto.sql.analyzer.ExpressionTreeUtils.tryResolveEnumLiteralType(ExpressionTreeUtils.java:149)
	at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:536)
	at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:389)

Do we have an example where this failure manifests?

@woshizhouyihao
Copy link
Contributor Author

woshizhouyihao commented Apr 23, 2025

@woshizhouyihao What's the reason for this revert?

We are seeing some errors internally as following, which are blocking some pipelines. Reverting those 2 commits resolve the issue. 3532782 68cdacd

at com.google.common.base.Preconditions.checkArgument(Preconditions.java:218)
	at com.facebook.presto.metadata.FunctionAndTypeManager.getUserDefinedType(FunctionAndTypeManager.java:761)
	at com.facebook.presto.metadata.FunctionAndTypeManager.getType(FunctionAndTypeManager.java:378)
	at com.facebook.presto.metadata.FunctionAndTypeManager$1.getType(FunctionAndTypeManager.java:200)
	at com.facebook.presto.sql.analyzer.ExpressionTreeUtils.tryResolveEnumLiteralType(ExpressionTreeUtils.java:149)
	at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:536)
	at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:389)
java.lang.IllegalArgumentException: QualifiedObjectName should have exactly 3 parts
	at com.facebook.presto.common.QualifiedObjectName.valueOf(QualifiedObjectName.java:45)
	at com.facebook.presto.common.type.TypeSignatureBase.of(TypeSignatureBase.java:38)
	at com.facebook.presto.common.type.TypeSignature.<init>(TypeSignature.java:101)
	at com.facebook.presto.common.type.TypeSignature.<init>(TypeSignature.java:96)
	at com.facebook.presto.common.type.TypeSignature.parseTypeSignature(TypeSignature.java:216)
	at com.facebook.presto.common.type.TypeSignature.parseTypeSignature(TypeSignature.java:195)
	at com.facebook.presto.sql.analyzer.ExpressionTreeUtils.tryResolveEnumLiteralType(ExpressionTreeUtils.java:149)
	at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:536)
	at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:389)

Do we have an example where this failure manifests?

here is one of the query texts

SELECT 
1 
FROM 
(
  SELECT 
    * 
  FROM 
    table_name
  WHERE 
    ds = '2025-04-20'
) p 
WHERE 
p.id2 = x.xxx.xxx.id
)

Here is the full error message:

java.lang.IllegalArgumentException: Cannot find function namespace for type 'x.xxx.xxx'
  at com.google.common.base.Preconditions.checkArgument(Preconditions.java:218)
  at com.facebook.presto.metadata.FunctionAndTypeManager.getUserDefinedType(FunctionAndTypeManager.java:761)
  at com.facebook.presto.metadata.FunctionAndTypeManager.getType(FunctionAndTypeManager.java:378)
  at com.facebook.presto.metadata.FunctionAndTypeManager$1.getType(FunctionAndTypeManager.java:200)
  at com.facebook.presto.sql.analyzer.ExpressionTreeUtils.tryResolveEnumLiteralType(ExpressionTreeUtils.java:149)
  at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:536)
  at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:389)
  at com.facebook.presto.sql.tree.DereferenceExpression.accept(DereferenceExpression.java:54)
  at com.facebook.presto.sql.tree.StackableAstVisitor.process(StackableAstVisitor.java:26)
  at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.process(ExpressionAnalyzer.java:412)
  at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.getOperator(ExpressionAnalyzer.java:1603)
  at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitComparisonExpression(ExpressionAnalyzer.java:608)
  at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitComparisonExpression(ExpressionAnalyzer.java:389)
  at com.facebook.presto.sql.tree.ComparisonExpression.accept(ComparisonExpression.java:71)
  at com.facebook.presto.sql.tree.StackableAstVisitor.process(StackableAstVisitor.java:26)
  at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.process(ExpressionAnalyzer.java:412)
  at com.facebook.presto.sql.analyzer.ExpressionAnalyzer.analyze(ExpressionAnalyzer.java:350)
  at com.facebook.presto.sql.analyzer.ExpressionAnalyzer.analyzeExpression(ExpressionAnalyzer.java:1941)
  at com.facebook.presto.sql.planner.TranslateExpressionsUtil.toRowExpression(TranslateExpressionsUtil.java:57)
  at com.facebook.presto.sql.planner.QueryPlanner.rowExpression(QueryPlanner.java:1327)
  at com.facebook.presto.sql.planner.QueryPlanner.filter(QueryPlanner.java:478)
  at com.facebook.presto.sql.planner.QueryPlanner.plan(QueryPlanner.java:208)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuerySpecification(RelationPlanner.java:776)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuerySpecification(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.QuerySpecification.accept(QuerySpecification.java:138)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.QueryPlanner.planQueryBody(QueryPlanner.java:411)
  at com.facebook.presto.sql.planner.QueryPlanner.plan(QueryPlanner.java:188)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuery(RelationPlanner.java:769)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuery(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.Query.accept(Query.java:105)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.DefaultTraversalVisitor.visitSubqueryExpression(DefaultTraversalVisitor.java:320)
  at com.facebook.presto.sql.tree.SubqueryExpression.accept(SubqueryExpression.java:51)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.SubqueryPlanner.createPlanBuilder(SubqueryPlanner.java:506)
  at com.facebook.presto.sql.planner.SubqueryPlanner.appendExistSubqueryApplyNode(SubqueryPlanner.java:301)
  at com.facebook.presto.sql.planner.SubqueryPlanner.appendExistsSubqueryApplyNodes(SubqueryPlanner.java:280)
  at com.facebook.presto.sql.planner.SubqueryPlanner.handleSubqueries(SubqueryPlanner.java:150)
  at com.facebook.presto.sql.planner.SubqueryPlanner.handleSubqueries(SubqueryPlanner.java:142)
  at com.facebook.presto.sql.planner.QueryPlanner.filter(QueryPlanner.java:475)
  at com.facebook.presto.sql.planner.QueryPlanner.plan(QueryPlanner.java:208)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuerySpecification(RelationPlanner.java:776)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuerySpecification(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.QuerySpecification.accept(QuerySpecification.java:138)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.QueryPlanner.planQueryBody(QueryPlanner.java:411)
  at com.facebook.presto.sql.planner.QueryPlanner.plan(QueryPlanner.java:188)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuery(RelationPlanner.java:769)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuery(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.Query.accept(Query.java:105)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.RelationPlanner.visitTable(RelationPlanner.java:187)
  at com.facebook.presto.sql.planner.RelationPlanner.visitTable(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.Table.accept(Table.java:60)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.RelationPlanner.visitAliasedRelation(RelationPlanner.java:231)
  at com.facebook.presto.sql.planner.RelationPlanner.visitAliasedRelation(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.AliasedRelation.accept(AliasedRelation.java:71)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.RelationPlanner.visitJoin(RelationPlanner.java:294)
  at com.facebook.presto.sql.planner.RelationPlanner.visitJoin(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.Join.accept(Join.java:90)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.QueryPlanner.planFrom(QueryPlanner.java:422)
  at com.facebook.presto.sql.planner.QueryPlanner.plan(QueryPlanner.java:205)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuerySpecification(RelationPlanner.java:776)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuerySpecification(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.QuerySpecification.accept(QuerySpecification.java:138)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.QueryPlanner.planQueryBody(QueryPlanner.java:411)
  at com.facebook.presto.sql.planner.QueryPlanner.plan(QueryPlanner.java:188)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuery(RelationPlanner.java:769)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuery(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.Query.accept(Query.java:105)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.RelationPlanner.visitTableSubquery(RelationPlanner.java:762)
  at com.facebook.presto.sql.planner.RelationPlanner.visitTableSubquery(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.TableSubquery.accept(TableSubquery.java:53)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.QueryPlanner.planFrom(QueryPlanner.java:422)
  at com.facebook.presto.sql.planner.QueryPlanner.plan(QueryPlanner.java:205)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuerySpecification(RelationPlanner.java:776)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuerySpecification(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.QuerySpecification.accept(QuerySpecification.java:138)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.QueryPlanner.planQueryBody(QueryPlanner.java:411)
  at com.facebook.presto.sql.planner.QueryPlanner.plan(QueryPlanner.java:188)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuery(RelationPlanner.java:769)
  at com.facebook.presto.sql.planner.RelationPlanner.visitQuery(RelationPlanner.java:137)
  at com.facebook.presto.sql.tree.Query.accept(Query.java:105)
  at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
  at com.facebook.presto.sql.planner.RelationPlanner.process(RelationPlanner.java:173)
  at com.facebook.presto.sql.planner.LogicalPlanner.createRelationPlan(LogicalPlanner.java:564)
  at com.facebook.presto.sql.planner.LogicalPlanner.buildInternalInsertPlan(LogicalPlanner.java:335)
  at com.facebook.presto.sql.planner.LogicalPlanner.createInsertPlan(LogicalPlanner.java:315)
  at com.facebook.presto.sql.planner.LogicalPlanner.planStatementWithoutOutput(LogicalPlanner.java:176)
  at com.facebook.presto.sql.planner.LogicalPlanner.planStatement(LogicalPlanner.java:160)
  at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:144)
  at com.facebook.presto.sql.analyzer.BuiltInQueryAnalyzer.plan(BuiltInQueryAnalyzer.java:107)
  at com.facebook.presto.execution.SqlQueryExecution.lambda$doCreateLogicalPlanAndOptimize$4(SqlQueryExecution.java:566)
  at com.facebook.presto.common.RuntimeStats.recordWallAndCpuTime(RuntimeStats.java:158)
  at com.facebook.presto.execution.SqlQueryExecution.doCreateLogicalPlanAndOptimize(SqlQueryExecution.java:564)
  at com.facebook.presto.common.RuntimeStats.recordWallAndCpuTime(RuntimeStats.java:158)
  at com.facebook.presto.execution.SqlQueryExecution.createLogicalPlanAndOptimize(SqlQueryExecution.java:551)
  at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:479)
  at com.facebook.presto.$gen.Presto_0_293_edge1_1_SNAPSHOT_$_git_commit_id_abbrev___0_293_edge1_1____20250418_175334_1.run(Unknown Source)
  at com.facebook.presto.execution.SqlQueryManager.createQuery(SqlQueryManager.java:320)
  at com.facebook.presto.dispatcher.LocalDispatchQuery.lambda$startExecution$8(LocalDispatchQuery.java:210)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:829)

@pdabre12
Copy link
Contributor

@woshizhouyihao Thanks for the stack trace.
Still trying to wrap my head around the failure, user defined types need to be created with a QualifiedObjectName, sample code attached below,

/**
 * UserDefinedType represents an enum type, or a distinct type.
 * Type definition is defined by user and is extracted at runtime.
 */
public class UserDefinedType
{
    private final QualifiedObjectName name;
    private final TypeSignature representation;

    public UserDefinedType(QualifiedObjectName name, TypeSignature representation)
    {
        this.name = requireNonNull(name, "name is null");
        this.representation = requireNonNull(representation, "representation is null");
    }

So how do we run into a situation where we don't have the fully qualified object name, is it possible to get more context on how/where these user defined types are created?

@woshizhouyihao
Copy link
Contributor Author

woshizhouyihao commented Apr 23, 2025

There are 2 exceptions above.
one fails for QualifiedObjectName

java.lang.IllegalArgumentException: QualifiedObjectName should have exactly 3 parts
	at com.facebook.presto.common.QualifiedObjectName.valueOf(QualifiedObjectName.java:45)

This is the code snippet where throws the error

        String[] parts = name.split("\\.");
        if (parts.length != 3) {
            throw new IllegalArgumentException("QualifiedObjectName should have exactly 3 parts");
        }

The other one fails for

java.lang.IllegalArgumentException: Cannot find function namespace for type 'x.xxx.xxx'
  at com.google.common.base.Preconditions.checkArgument(Preconditions.java:218)
  at com.facebook.presto.metadata.FunctionAndTypeManager.getUserDefinedType(FunctionAndTypeManager.java:761)

It seems something is wrong with type parsing.

The error is thrown during query text parsing. Something is wrong here:

return new TypeSignatureBase(QualifiedObjectName.valueOf(name.toLowerCase(ENGLISH)));

@pdabre12
Copy link
Contributor

@woshizhouyihao So, to sum it up, you are running into both of these exceptions,
when running this query,

SELECT 
1 
FROM 
(
  SELECT 
    * 
  FROM 
    table_name
  WHERE 
    ds = '2025-04-20'
) p 
WHERE 
p.id2 = x.xxx.xxx.id
)

@pdabre12
Copy link
Contributor

Please hold on to merging this, I am trying to come up with a fix.

@woshizhouyihao
Copy link
Contributor Author

will merge #24981 for the fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants